File System Summary =================== WARN: Device /dev/loop0 (/snap/core18/1997) is 100% full, might be an issue WARN: Device /dev/loop1 (/snap/lxd/19647) is 100% full, might be an issue WARN: Device /dev/loop2 (/snap/snapd/11588) is 100% full, might be an issue Filesystem Size Used Avail Use% Mounted on udev 5.9G 0 5.9G 0% /dev tmpfs 1.2G 1.2M 1.2G 1% /run /dev/sda1 39G 4.9G 34G 13% / tmpfs 5.9G 0 5.9G 0% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 5.9G 0 5.9G 0% /sys/fs/cgroup /dev/loop0 56M 56M 0 100% /snap/core18/1997 /dev/loop1 71M 71M 0 100% /snap/lxd/19647 /dev/loop2 33M 33M 0 100% /snap/snapd/11588 /dev/sda15 105M 7.8M 97M 8% /boot/efi tmpfs 1.2G 0 1.2G 0% /run/user/1000 tmpfs 512M 0 512M 0% /opt/stack/data/etcd Process Listing =============== ps axo user,ppid,pid,pcpu,pmem,vsz,rss,tty,stat,start,time,args --------------------------------------------------------------- USER PPID PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND root 0 1 1.8 0.1 168964 12992 ? Ss 15:10:06 00:00:34 /lib/systemd/systemd --system --deserialize 27 root 0 2 0.0 0.0 0 0 ? S 15:10:06 00:00:00 [kthreadd] root 2 3 0.0 0.0 0 0 ? I< 15:10:06 00:00:00 [rcu_gp] root 2 4 0.0 0.0 0 0 ? I< 15:10:06 00:00:00 [rcu_par_gp] root 2 6 0.0 0.0 0 0 ? I< 15:10:06 00:00:00 [kworker/0:0H-kblockd] root 2 8 0.0 0.0 0 0 ? I< 15:10:06 00:00:00 [mm_percpu_wq] root 2 9 0.0 0.0 0 0 ? S 15:10:06 00:00:00 [ksoftirqd/0] root 2 10 0.0 0.0 0 0 ? I 15:10:06 00:00:01 [rcu_sched] root 2 11 0.0 0.0 0 0 ? S 15:10:06 00:00:00 [migration/0] root 2 12 0.0 0.0 0 0 ? S 15:10:06 00:00:00 [idle_inject/0] root 2 14 0.0 0.0 0 0 ? S 15:10:06 00:00:00 [cpuhp/0] root 2 15 0.0 0.0 0 0 ? S 15:10:06 00:00:00 [cpuhp/1] root 2 16 0.0 0.0 0 0 ? S 15:10:06 00:00:00 [idle_inject/1] root 2 17 0.0 0.0 0 0 ? S 15:10:06 00:00:00 [migration/1] root 2 18 0.0 0.0 0 0 ? S 15:10:06 00:00:00 [ksoftirqd/1] root 2 20 0.0 0.0 0 0 ? I< 15:10:06 00:00:00 [kworker/1:0H-kblockd] root 2 21 0.0 0.0 0 0 ? S 15:10:06 00:00:00 [cpuhp/2] root 2 22 0.0 0.0 0 0 ? S 15:10:06 00:00:00 [idle_inject/2] root 2 23 0.0 0.0 0 0 ? S 15:10:06 00:00:00 [migration/2] root 2 24 0.0 0.0 0 0 ? S 15:10:06 00:00:00 [ksoftirqd/2] root 2 26 0.0 0.0 0 0 ? I< 15:10:06 00:00:00 [kworker/2:0H-kblockd] root 2 27 0.0 0.0 0 0 ? S 15:10:06 00:00:00 [cpuhp/3] root 2 28 0.0 0.0 0 0 ? S 15:10:06 00:00:00 [idle_inject/3] root 2 29 0.0 0.0 0 0 ? S 15:10:06 00:00:00 [migration/3] root 2 30 0.0 0.0 0 0 ? S 15:10:06 00:00:00 [ksoftirqd/3] root 2 32 0.0 0.0 0 0 ? I< 15:10:06 00:00:00 [kworker/3:0H-kblockd] root 2 33 0.0 0.0 0 0 ? S 15:10:06 00:00:00 [kdevtmpfs] root 2 34 0.0 0.0 0 0 ? I< 15:10:06 00:00:00 [netns] root 2 35 0.0 0.0 0 0 ? S 15:10:06 00:00:00 [rcu_tasks_kthre] root 2 36 0.0 0.0 0 0 ? S 15:10:06 00:00:00 [kauditd] root 2 40 0.0 0.0 0 0 ? S 15:10:06 00:00:00 [khungtaskd] root 2 41 0.0 0.0 0 0 ? S 15:10:06 00:00:00 [oom_reaper] root 2 42 0.0 0.0 0 0 ? I< 15:10:06 00:00:00 [writeback] root 2 43 0.0 0.0 0 0 ? S 15:10:06 00:00:00 [kcompactd0] root 2 44 0.0 0.0 0 0 ? SN 15:10:06 00:00:00 [ksmd] root 2 45 0.0 0.0 0 0 ? SN 15:10:06 00:00:00 [khugepaged] root 2 91 0.0 0.0 0 0 ? I< 15:10:06 00:00:00 [kintegrityd] root 2 92 0.0 0.0 0 0 ? I< 15:10:06 00:00:00 [kblockd] root 2 93 0.0 0.0 0 0 ? I< 15:10:06 00:00:00 [blkcg_punt_bio] root 2 94 0.0 0.0 0 0 ? I< 15:10:06 00:00:00 [tpm_dev_wq] root 2 95 0.0 0.0 0 0 ? I< 15:10:06 00:00:00 [ata_sff] root 2 96 0.0 0.0 0 0 ? I< 15:10:07 00:00:00 [md] root 2 97 0.0 0.0 0 0 ? I< 15:10:07 00:00:00 [edac-poller] root 2 98 0.0 0.0 0 0 ? I< 15:10:07 00:00:00 [devfreq_wq] root 2 99 0.0 0.0 0 0 ? S 15:10:07 00:00:00 [watchdogd] root 2 102 0.0 0.0 0 0 ? S 15:10:07 00:00:00 [kswapd0] root 2 103 0.0 0.0 0 0 ? S 15:10:07 00:00:00 [ecryptfs-kthrea] root 2 105 0.0 0.0 0 0 ? I< 15:10:07 00:00:00 [kthrotld] root 2 106 0.0 0.0 0 0 ? I< 15:10:07 00:00:00 [acpi_thermal_pm] root 2 109 0.0 0.0 0 0 ? S 15:10:08 00:00:00 [scsi_eh_0] root 2 110 0.0 0.0 0 0 ? I< 15:10:08 00:00:00 [scsi_tmf_0] root 2 111 0.0 0.0 0 0 ? S 15:10:08 00:00:00 [scsi_eh_1] root 2 112 0.0 0.0 0 0 ? I< 15:10:08 00:00:00 [scsi_tmf_1] root 2 114 0.0 0.0 0 0 ? I< 15:10:08 00:00:00 [vfio-irqfd-clea] root 2 115 0.0 0.0 0 0 ? I< 15:10:08 00:00:00 [ipv6_addrconf] root 2 116 0.0 0.0 0 0 ? I 15:10:08 00:00:01 [kworker/u256:3-events_power_efficient] root 2 125 0.0 0.0 0 0 ? I< 15:10:08 00:00:00 [kstrp] root 2 129 0.0 0.0 0 0 ? I< 15:10:09 00:00:00 [kworker/u257:0] root 2 142 0.0 0.0 0 0 ? I< 15:10:09 00:00:00 [charger_manager] root 2 185 0.0 0.0 0 0 ? I< 15:10:09 00:00:00 [hv_vmbus_con] root 2 186 0.0 0.0 0 0 ? I< 15:10:09 00:00:00 [hv_pri_chan] root 2 187 0.0 0.0 0 0 ? I< 15:10:09 00:00:00 [kworker/3:1H-kblockd] root 2 188 0.0 0.0 0 0 ? I< 15:10:09 00:00:00 [hv_sub_chan] root 2 189 0.0 0.0 0 0 ? I 15:10:09 00:00:00 [kworker/2:3-events] root 2 190 0.0 0.0 0 0 ? I< 15:10:09 00:00:00 [cryptd] root 2 231 0.0 0.0 0 0 ? I< 15:10:09 00:00:00 [kworker/2:1H-kblockd] root 2 233 0.0 0.0 0 0 ? S 15:10:09 00:00:00 [scsi_eh_2] root 2 234 0.0 0.0 0 0 ? S 15:10:09 00:00:00 [scsi_eh_3] root 2 235 0.0 0.0 0 0 ? I< 15:10:09 00:00:00 [scsi_tmf_2] root 2 236 0.0 0.0 0 0 ? I< 15:10:09 00:00:00 [scsi_tmf_3] root 2 237 0.0 0.0 0 0 ? I< 15:10:09 00:00:00 [storvsc_error_w] root 2 238 0.0 0.0 0 0 ? I< 15:10:09 00:00:00 [storvsc_error_w] root 2 251 0.0 0.0 0 0 ? I< 15:10:09 00:00:00 [kworker/0:1H-kblockd] root 2 278 0.0 0.0 0 0 ? I< 15:10:11 00:00:00 [raid5wq] root 2 304 0.0 0.0 0 0 ? I< 15:10:11 00:00:00 [kworker/1:1H-kblockd] root 2 319 0.0 0.0 0 0 ? S 15:10:11 00:00:01 [jbd2/sda1-8] root 2 320 0.0 0.0 0 0 ? I< 15:10:11 00:00:00 [ext4-rsv-conver] root 1 424 0.0 0.0 18944 5556 ? Ss 15:10:14 00:00:01 /lib/systemd/systemd-udevd root 2 449 0.0 0.0 0 0 ? S 15:10:14 00:00:00 [hv_balloon] root 2 551 0.0 0.0 0 0 ? I< 15:10:14 00:00:00 [kaluad] root 2 552 0.0 0.0 0 0 ? I< 15:10:14 00:00:00 [kmpath_rdacd] root 2 553 0.0 0.0 0 0 ? I< 15:10:14 00:00:00 [kmpathd] root 2 554 0.0 0.0 0 0 ? I< 15:10:14 00:00:00 [kmpath_handlerd] root 1 555 0.0 0.1 280136 17940 ? SLsl 15:10:14 00:00:00 /sbin/multipathd -d -s root 2 563 0.0 0.0 0 0 ? S< 15:10:14 00:00:00 [loop0] root 2 567 0.0 0.0 0 0 ? S< 15:10:14 00:00:00 [loop1] root 2 569 0.0 0.0 0 0 ? S< 15:10:14 00:00:00 [loop2] root 1 769 0.0 0.0 241048 9308 ? Ssl 15:10:26 00:00:00 /usr/lib/accountsservice/accounts-daemon root 1 772 0.0 0.0 8536 2992 ? Ss 15:10:26 00:00:00 /usr/sbin/cron -f message+ 1 775 0.1 0.0 7776 4964 ? Ss 15:10:26 00:00:03 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only root 1 784 0.0 0.0 81828 3636 ? Ssl 15:10:26 00:00:00 /usr/sbin/irqbalance --foreground root 1 791 0.0 0.1 29264 17908 ? Ss 15:10:26 00:00:00 /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers root 1 795 0.1 0.2 1222896 35788 ? Ssl 15:10:27 00:00:03 /usr/lib/snapd/snapd root 1 796 0.1 0.0 16780 7668 ? Ss 15:10:27 00:00:01 /lib/systemd/systemd-logind daemon 1 798 0.0 0.0 3792 2228 ? Ss 15:10:27 00:00:00 /usr/sbin/atd -f root 1 831 0.0 0.0 7352 2224 ttyS0 Ss+ 15:10:27 00:00:00 /sbin/agetty -o -p -- \u --keep-baud 115200,38400,9600 ttyS0 vt220 root 1 850 0.0 0.0 5828 1780 tty1 Ss+ 15:10:27 00:00:00 /sbin/agetty -o -p -- \u --noclear tty1 linux root 1 861 0.0 0.1 108100 20644 ? Ssl 15:10:27 00:00:00 /usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal root 1 870 0.0 0.0 237756 10456 ? Ssl 15:10:27 00:00:00 /usr/lib/policykit-1/polkitd --no-debug root 1 1562 0.0 0.0 13796 9116 ? Ss 15:14:02 00:00:00 sshd: ubuntu [priv] ubuntu 1 1578 0.0 0.0 18520 9648 ? Ss 15:14:02 00:00:00 /lib/systemd/systemd --user ubuntu 1578 1579 0.0 0.0 104236 4528 ? S 15:14:02 00:00:00 (sd-pam) ubuntu 1562 1655 0.0 0.0 13928 6576 ? S 15:14:03 00:00:00 sshd: ubuntu@pts/0 root 2 8551 0.0 0.0 0 0 ? I 15:21:34 00:00:00 [kworker/3:1-events] ubuntu 1655 8982 0.0 0.0 2608 536 pts/0 Ss+ 15:23:59 00:00:00 /bin/sh -c /usr/bin/python3 /home/ubuntu/.ansible/tmp/ansible-tmp-1632237840.33-115696845548812/AnsiballZ_command.py && sleep 0 ubuntu 8982 8983 0.0 0.1 27828 18060 pts/0 S+ 15:23:59 00:00:00 /usr/bin/python3 /home/ubuntu/.ansible/tmp/ansible-tmp-1632237840.33-115696845548812/AnsiballZ_command.py ubuntu 8983 8985 0.0 0.0 2608 608 pts/0 S+ 15:23:59 00:00:00 /bin/sh -c ./unstack.sh > /dev/null 2>&1 ./stack.sh > /dev/null 2>&1 root 2 10192 0.0 0.0 0 0 ? I 15:24:02 00:00:00 [kworker/u256:0-events_power_efficient] root 2 10193 0.0 0.0 2488 592 ? S 15:24:02 00:00:00 bpfilter_umh root 2 11359 0.0 0.0 0 0 ? I 15:24:04 00:00:00 [kworker/1:1-events] ubuntu 8985 11726 0.3 0.0 14752 10124 pts/0 S+ 15:24:06 00:00:03 bash ./stack.sh ubuntu 11726 13985 0.0 0.0 10644 3888 pts/0 S+ 15:24:09 00:00:00 bash ./stack.sh ubuntu 11726 13990 0.0 0.0 10644 3888 pts/0 S+ 15:24:09 00:00:00 bash ./stack.sh ubuntu 13985 13991 0.1 0.0 17384 11244 pts/0 S+ 15:24:09 00:00:01 /usr/bin/python3.8 /home/ubuntu/devstack/tools/outfilter.py -v --no-timestamp -o /opt/stack/logs/stack.sh.log.2021-09-21-152410 ubuntu 13990 13997 0.0 0.0 17384 11380 pts/0 S+ 15:24:09 00:00:00 /usr/bin/python3.8 /home/ubuntu/devstack/tools/outfilter.py -o /opt/stack/logs/stack.sh.log.2021-09-21-152410.summary.2021-09-21-152410 memcache 1 18808 0.0 0.0 409768 3868 ? Ssl 15:27:24 00:00:00 /usr/bin/memcached -m 64 -p 11211 -u memcache -l 127.0.0.1 -P /var/run/memcached/memcached.pid root 2 18986 0.0 0.0 0 0 ? I 15:27:31 00:00:00 [kworker/0:3-rcu_par_gp] root 1 19068 0.0 0.0 7744 2984 ? Ss 15:27:40 00:00:00 /usr/sbin/conntrackd -C /etc/conntrackd/conntrackd.conf root 1 19233 0.0 0.0 15304 9068 ? Ss 15:27:50 00:00:00 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock haproxy 19233 19243 0.0 0.0 236500 3004 ? Sl 15:27:50 00:00:00 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock root 2 20449 0.0 0.0 0 0 ? I 15:28:00 00:00:00 [kworker/3:3-memcg_kmem_cache] root 1 20771 0.0 0.0 12176 7008 ? Ss 15:28:10 00:00:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups pcp 1 21896 0.3 0.0 8360 4420 ? Ss 15:28:40 00:00:02 /usr/lib/pcp/bin/pmcd root 21896 21899 0.0 0.0 7736 3828 ? S 15:28:40 00:00:00 /var/lib/pcp/pmdas/root/pmdaroot root 21899 21900 0.0 0.0 8308 5036 ? S 15:28:40 00:00:00 /var/lib/pcp/pmdas/proc/pmdaproc -d 3 root 21899 21901 0.0 0.0 7212 1640 ? S 15:28:40 00:00:00 /var/lib/pcp/pmdas/xfs/pmdaxfs -d 11 root 21899 21902 0.1 0.0 8272 5044 ? S 15:28:40 00:00:00 /var/lib/pcp/pmdas/linux/pmdalinux root 21899 21903 0.0 0.0 7272 3812 ? S 15:28:40 00:00:00 /var/lib/pcp/pmdas/kvm/pmdakvm -d 95 systemd+ 1 33147 0.0 0.0 34932 7948 ? Ss 15:28:58 00:00:00 /lib/systemd/systemd-networkd systemd+ 1 33159 0.0 0.0 23896 12216 ? Ss 15:28:58 00:00:00 /lib/systemd/systemd-resolved root 2 33160 0.0 0.0 0 0 ? I 15:28:58 00:00:00 [kworker/3:4-events] systemd+ 1 33287 0.0 0.0 90228 6152 ? Ssl 15:29:00 00:00:00 /lib/systemd/systemd-timesyncd syslog 1 38000 0.0 0.0 224348 4336 ? Ssl 15:29:16 00:00:00 /usr/sbin/rsyslogd -n -iNONE root 1 45647 0.0 0.1 283808 16132 ? Ssl 15:29:46 00:00:00 /usr/lib/packagekit/packagekitd ubuntu 1578 47076 0.0 0.0 7108 4148 ? Ss 15:29:57 00:00:00 /usr/bin/dbus-daemon --session --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only root 2 47564 0.0 0.0 0 0 ? I 15:30:07 00:00:00 [kworker/u256:1-events_power_efficient] root 1 48189 0.0 0.0 37040 10504 ? S mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: mtu 1500 qdisc mq state UP group default qlen 1000 link/ether fa:16:3e:b3:fa:90 brd ff:ff:ff:ff:ff:ff inet 10.222.0.36/22 brd 10.222.3.255 scope global dynamic eth0 valid_lft 85709sec preferred_lft 85709sec inet6 fe80::f816:3eff:feb3:fa90/64 scope link valid_lft forever preferred_lft forever 3: eth1: mtu 1500 qdisc mq state UP group default qlen 1000 link/ether fa:16:3e:fb:15:e6 brd ff:ff:ff:ff:ff:ff inet 192.168.0.28/22 brd 192.168.3.255 scope global dynamic eth1 valid_lft 85709sec preferred_lft 85709sec inet6 fe80::f816:3eff:fefb:15e6/64 scope link valid_lft forever preferred_lft forever ip route -------- default via 10.222.0.1 dev eth0 proto dhcp src 10.222.0.36 metric 100 10.222.0.0/22 dev eth0 proto kernel scope link src 10.222.0.36 10.222.0.0/22 dev eth1 proto dhcp scope link src 192.168.0.28 metric 100 169.254.169.254 via 10.222.0.12 dev eth0 proto dhcp src 10.222.0.36 metric 100 169.254.169.254 via 192.168.0.12 dev eth1 proto dhcp src 192.168.0.28 metric 100 192.168.0.0/22 dev eth1 proto kernel scope link src 192.168.0.28 192.168.0.0/22 dev eth0 proto dhcp scope link src 10.222.0.36 metric 100 ip -6 route ----------- ::1 dev lo proto kernel metric 256 pref medium fe80::/64 dev eth1 proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium ip netns -------- Open vSwitch Dump ================= sudo ovs-vsctl show ------------------- cf10b959-2c46-431e-8400-892ca416c243 ovs_version: "2.13.3" IP Tables Dump ============== sudo iptables --line-numbers -L -nv -t filter --------------------------------------------- Chain INPUT (policy ACCEPT 93826 packets, 330M bytes) num pkts bytes target prot opt in out source destination Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) num pkts bytes target prot opt in out source destination Chain OUTPUT (policy ACCEPT 41958 packets, 2860K bytes) num pkts bytes target prot opt in out source destination sudo iptables --line-numbers -L -nv -t nat ------------------------------------------ Chain PREROUTING (policy ACCEPT 974 packets, 276K bytes) num pkts bytes target prot opt in out source destination Chain INPUT (policy ACCEPT 784 packets, 252K bytes) num pkts bytes target prot opt in out source destination Chain OUTPUT (policy ACCEPT 560 packets, 47966 bytes) num pkts bytes target prot opt in out source destination Chain POSTROUTING (policy ACCEPT 560 packets, 47966 bytes) num pkts bytes target prot opt in out source destination sudo iptables --line-numbers -L -nv -t mangle --------------------------------------------- Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes) num pkts bytes target prot opt in out source destination Chain INPUT (policy ACCEPT 0 packets, 0 bytes) num pkts bytes target prot opt in out source destination Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) num pkts bytes target prot opt in out source destination Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes) num pkts bytes target prot opt in out source destination Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes) num pkts bytes target prot opt in out source destination EB Tables Dump ============== sudo ebtables -t filter -L -------------------------- Bridge table: filter Bridge chain: INPUT, entries: 0, policy: ACCEPT Bridge chain: FORWARD, entries: 0, policy: ACCEPT Bridge chain: OUTPUT, entries: 0, policy: ACCEPT sudo ebtables -t nat -L ----------------------- Bridge table: nat Bridge chain: PREROUTING, entries: 0, policy: ACCEPT Bridge chain: OUTPUT, entries: 0, policy: ACCEPT Bridge chain: POSTROUTING, entries: 0, policy: ACCEPT Compute consoles ================ nova-compute Guru Meditation Report =================================== Skipping as nova-compute does not appear to be running neutron-dhcp-agent Guru Meditation Report ========================================= Skipping as neutron-dhcp-agent does not appear to be running neutron-l3-agent Guru Meditation Report ======================================= Skipping as neutron-l3-agent does not appear to be running neutron-linuxbridge-agent Guru Meditation Report ================================================ Skipping as neutron-linuxbridge-agent does not appear to be running neutron-metadata-agent Guru Meditation Report ============================================= Skipping as neutron-metadata-agent does not appear to be running neutron-openvswitch-agent Guru Meditation Report ================================================ Skipping as neutron-openvswitch-agent does not appear to be running cinder-volume Guru Meditation Report ==================================== Skipping as cinder-volume does not appear to be running