File System Summary =================== WARN: Device /dev/loop1 (/snap/snapd/11588) is 100% full, might be an issue WARN: Device /dev/loop0 (/snap/core18/1997) is 100% full, might be an issue WARN: Device /dev/loop2 (/snap/lxd/19647) is 100% full, might be an issue Filesystem Size Used Avail Use% Mounted on udev 5.9G 0 5.9G 0% /dev tmpfs 1.2G 1.2M 1.2G 1% /run /dev/sda1 39G 5.2G 34G 14% / tmpfs 5.9G 0 5.9G 0% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 5.9G 0 5.9G 0% /sys/fs/cgroup /dev/loop1 33M 33M 0 100% /snap/snapd/11588 /dev/loop0 56M 56M 0 100% /snap/core18/1997 /dev/loop2 71M 71M 0 100% /snap/lxd/19647 /dev/sda15 105M 7.8M 97M 8% /boot/efi tmpfs 1.2G 0 1.2G 0% /run/user/1000 tmpfs 512M 0 512M 0% /opt/stack/data/etcd Process Listing =============== ps axo user,ppid,pid,pcpu,pmem,vsz,rss,tty,stat,start,time,args --------------------------------------------------------------- USER PPID PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND root 0 1 2.0 0.1 168972 13132 ? Ss 20:08:47 00:00:33 /lib/systemd/systemd --system --deserialize 27 root 0 2 0.0 0.0 0 0 ? S 20:08:47 00:00:00 [kthreadd] root 2 3 0.0 0.0 0 0 ? I< 20:08:47 00:00:00 [rcu_gp] root 2 4 0.0 0.0 0 0 ? I< 20:08:47 00:00:00 [rcu_par_gp] root 2 6 0.0 0.0 0 0 ? I< 20:08:47 00:00:00 [kworker/0:0H-kblockd] root 2 7 0.0 0.0 0 0 ? I 20:08:47 00:00:01 [kworker/u256:0-events_power_efficient] root 2 8 0.0 0.0 0 0 ? I< 20:08:47 00:00:00 [mm_percpu_wq] root 2 9 0.0 0.0 0 0 ? S 20:08:47 00:00:00 [ksoftirqd/0] root 2 10 0.0 0.0 0 0 ? I 20:08:47 00:00:01 [rcu_sched] root 2 11 0.0 0.0 0 0 ? S 20:08:47 00:00:00 [migration/0] root 2 12 0.0 0.0 0 0 ? S 20:08:47 00:00:00 [idle_inject/0] root 2 14 0.0 0.0 0 0 ? S 20:08:47 00:00:00 [cpuhp/0] root 2 15 0.0 0.0 0 0 ? S 20:08:47 00:00:00 [cpuhp/1] root 2 16 0.0 0.0 0 0 ? S 20:08:47 00:00:00 [idle_inject/1] root 2 17 0.0 0.0 0 0 ? S 20:08:47 00:00:00 [migration/1] root 2 18 0.0 0.0 0 0 ? S 20:08:47 00:00:00 [ksoftirqd/1] root 2 20 0.0 0.0 0 0 ? I< 20:08:47 00:00:00 [kworker/1:0H-kblockd] root 2 21 0.0 0.0 0 0 ? S 20:08:47 00:00:00 [cpuhp/2] root 2 22 0.0 0.0 0 0 ? S 20:08:47 00:00:00 [idle_inject/2] root 2 23 0.0 0.0 0 0 ? S 20:08:47 00:00:00 [migration/2] root 2 24 0.0 0.0 0 0 ? S 20:08:47 00:00:00 [ksoftirqd/2] root 2 26 0.0 0.0 0 0 ? I< 20:08:47 00:00:00 [kworker/2:0H-kblockd] root 2 27 0.0 0.0 0 0 ? S 20:08:47 00:00:00 [cpuhp/3] root 2 28 0.0 0.0 0 0 ? S 20:08:47 00:00:00 [idle_inject/3] root 2 29 0.0 0.0 0 0 ? S 20:08:47 00:00:00 [migration/3] root 2 30 0.0 0.0 0 0 ? S 20:08:47 00:00:00 [ksoftirqd/3] root 2 32 0.0 0.0 0 0 ? I< 20:08:47 00:00:00 [kworker/3:0H-kblockd] root 2 33 0.0 0.0 0 0 ? S 20:08:47 00:00:00 [kdevtmpfs] root 2 34 0.0 0.0 0 0 ? I< 20:08:47 00:00:00 [netns] root 2 35 0.0 0.0 0 0 ? S 20:08:47 00:00:00 [rcu_tasks_kthre] root 2 36 0.0 0.0 0 0 ? S 20:08:47 00:00:00 [kauditd] root 2 38 0.0 0.0 0 0 ? I 20:08:47 00:00:00 [kworker/1:1-cgroup_destroy] root 2 41 0.0 0.0 0 0 ? I 20:08:47 00:00:00 [kworker/2:2-rcu_par_gp] root 2 42 0.0 0.0 0 0 ? S 20:08:47 00:00:00 [khungtaskd] root 2 43 0.0 0.0 0 0 ? S 20:08:47 00:00:00 [oom_reaper] root 2 44 0.0 0.0 0 0 ? I< 20:08:47 00:00:00 [writeback] root 2 45 0.0 0.0 0 0 ? S 20:08:47 00:00:00 [kcompactd0] root 2 46 0.0 0.0 0 0 ? SN 20:08:47 00:00:00 [ksmd] root 2 47 0.0 0.0 0 0 ? SN 20:08:47 00:00:00 [khugepaged] root 2 94 0.0 0.0 0 0 ? I< 20:08:47 00:00:00 [kintegrityd] root 2 95 0.0 0.0 0 0 ? I< 20:08:47 00:00:00 [kblockd] root 2 96 0.0 0.0 0 0 ? I< 20:08:47 00:00:00 [blkcg_punt_bio] root 2 97 0.0 0.0 0 0 ? I< 20:08:47 00:00:00 [tpm_dev_wq] root 2 98 0.0 0.0 0 0 ? I< 20:08:47 00:00:00 [ata_sff] root 2 99 0.0 0.0 0 0 ? I< 20:08:48 00:00:00 [md] root 2 100 0.0 0.0 0 0 ? I< 20:08:48 00:00:00 [edac-poller] root 2 101 0.0 0.0 0 0 ? I< 20:08:48 00:00:00 [devfreq_wq] root 2 102 0.0 0.0 0 0 ? S 20:08:48 00:00:00 [watchdogd] root 2 105 0.0 0.0 0 0 ? S 20:08:48 00:00:00 [kswapd0] root 2 106 0.0 0.0 0 0 ? S 20:08:48 00:00:00 [ecryptfs-kthrea] root 2 108 0.0 0.0 0 0 ? I< 20:08:48 00:00:00 [kthrotld] root 2 109 0.0 0.0 0 0 ? I< 20:08:48 00:00:00 [acpi_thermal_pm] root 2 110 0.0 0.0 0 0 ? S 20:08:48 00:00:00 [scsi_eh_0] root 2 111 0.0 0.0 0 0 ? I< 20:08:48 00:00:00 [scsi_tmf_0] root 2 112 0.0 0.0 0 0 ? S 20:08:48 00:00:00 [scsi_eh_1] root 2 113 0.0 0.0 0 0 ? I< 20:08:48 00:00:00 [scsi_tmf_1] root 2 115 0.0 0.0 0 0 ? I< 20:08:48 00:00:00 [vfio-irqfd-clea] root 2 116 0.0 0.0 0 0 ? I< 20:08:49 00:00:00 [ipv6_addrconf] root 2 126 0.0 0.0 0 0 ? I< 20:08:49 00:00:00 [kstrp] root 2 129 0.0 0.0 0 0 ? I< 20:08:49 00:00:00 [kworker/u257:0] root 2 142 0.0 0.0 0 0 ? I< 20:08:49 00:00:00 [charger_manager] root 2 181 0.0 0.0 0 0 ? I< 20:08:50 00:00:00 [hv_vmbus_con] root 2 182 0.0 0.0 0 0 ? I< 20:08:50 00:00:00 [hv_pri_chan] root 2 183 0.0 0.0 0 0 ? I< 20:08:50 00:00:00 [hv_sub_chan] root 2 184 0.0 0.0 0 0 ? I< 20:08:50 00:00:00 [kworker/3:1H-kblockd] root 2 185 0.0 0.0 0 0 ? I< 20:08:50 00:00:00 [cryptd] root 2 223 0.0 0.0 0 0 ? S 20:08:50 00:00:00 [scsi_eh_2] root 2 224 0.0 0.0 0 0 ? S 20:08:50 00:00:00 [scsi_eh_3] root 2 225 0.0 0.0 0 0 ? I< 20:08:50 00:00:00 [scsi_tmf_2] root 2 226 0.0 0.0 0 0 ? I< 20:08:50 00:00:00 [scsi_tmf_3] root 2 227 0.0 0.0 0 0 ? I< 20:08:50 00:00:00 [storvsc_error_w] root 2 228 0.0 0.0 0 0 ? I< 20:08:50 00:00:00 [storvsc_error_w] root 2 240 0.0 0.0 0 0 ? I< 20:08:50 00:00:00 [kworker/0:1H-kblockd] root 2 275 0.0 0.0 0 0 ? I< 20:08:52 00:00:00 [raid5wq] root 2 301 0.0 0.0 0 0 ? I< 20:08:52 00:00:00 [kworker/1:1H-kblockd] root 2 302 0.0 0.0 0 0 ? I< 20:08:52 00:00:00 [kworker/2:1H-kblockd] root 2 317 0.0 0.0 0 0 ? S 20:08:52 00:00:01 [jbd2/sda1-8] root 2 318 0.0 0.0 0 0 ? I< 20:08:52 00:00:00 [ext4-rsv-conver] root 1 422 0.0 0.0 19356 5888 ? Ss 20:08:54 00:00:01 /lib/systemd/systemd-udevd root 2 451 0.0 0.0 0 0 ? S 20:08:55 00:00:00 [hv_balloon] root 2 551 0.0 0.0 0 0 ? I< 20:08:55 00:00:00 [kaluad] root 2 552 0.0 0.0 0 0 ? I< 20:08:55 00:00:00 [kmpath_rdacd] root 2 553 0.0 0.0 0 0 ? I< 20:08:55 00:00:00 [kmpathd] root 2 554 0.0 0.0 0 0 ? I< 20:08:55 00:00:00 [kmpath_handlerd] root 1 555 0.0 0.1 280136 17940 ? SLsl 20:08:55 00:00:00 /sbin/multipathd -d -s root 2 564 0.0 0.0 0 0 ? S< 20:08:55 00:00:00 [loop0] root 2 567 0.0 0.0 0 0 ? S< 20:08:55 00:00:00 [loop1] root 2 569 0.0 0.0 0 0 ? S< 20:08:55 00:00:00 [loop2] root 1 785 0.0 0.0 241048 9632 ? Ssl 20:09:06 00:00:00 /usr/lib/accountsservice/accounts-daemon root 1 789 0.0 0.0 8536 2880 ? Ss 20:09:06 00:00:00 /usr/sbin/cron -f message+ 1 791 0.2 0.0 7732 4908 ? Ss 20:09:06 00:00:03 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only root 1 798 0.0 0.0 81828 3620 ? Ssl 20:09:06 00:00:00 /usr/sbin/irqbalance --foreground root 1 799 0.0 0.1 29264 17896 ? Ss 20:09:06 00:00:00 /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers root 1 805 0.1 0.2 1446204 34936 ? Ssl 20:09:06 00:00:02 /usr/lib/snapd/snapd root 1 807 0.1 0.0 16780 7700 ? Ss 20:09:06 00:00:01 /lib/systemd/systemd-logind daemon 1 809 0.0 0.0 3792 2292 ? Ss 20:09:06 00:00:00 /usr/sbin/atd -f root 1 853 0.0 0.0 7352 2288 ttyS0 Ss+ 20:09:06 00:00:00 /sbin/agetty -o -p -- \u --keep-baud 115200,38400,9600 ttyS0 vt220 root 1 876 0.0 0.0 5828 1916 tty1 Ss+ 20:09:06 00:00:00 /sbin/agetty -o -p -- \u --noclear tty1 linux root 1 878 0.0 0.1 108100 20972 ? Ssl 20:09:06 00:00:00 /usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal root 1 885 0.0 0.0 237752 10480 ? Ssl 20:09:06 00:00:00 /usr/lib/policykit-1/polkitd --no-debug root 1 1582 0.0 0.0 13800 9064 ? Ss 20:13:45 00:00:00 sshd: ubuntu [priv] ubuntu 1 1598 0.0 0.0 18488 9764 ? Ss 20:13:45 00:00:00 /lib/systemd/systemd --user ubuntu 1598 1600 0.0 0.0 169792 4556 ? S 20:13:45 00:00:00 (sd-pam) ubuntu 1582 1680 0.0 0.0 13932 6580 ? S 20:13:46 00:00:00 sshd: ubuntu@pts/0 root 2 3156 0.0 0.0 0 0 ? I 20:14:58 00:00:00 [kworker/1:0-events] root 2 14376 0.0 0.0 0 0 ? I 20:20:56 00:00:00 [kworker/3:0-memcg_kmem_cache] root 2 14389 0.0 0.0 0 0 ? I 20:20:56 00:00:00 [kworker/3:3-events] root 2 14607 0.0 0.0 0 0 ? I 20:22:59 00:00:00 [kworker/u256:1-events_power_efficient] root 2 14724 0.0 0.0 0 0 ? I 20:24:04 00:00:00 [kworker/0:1-rcu_par_gp] root 2 14726 0.0 0.0 0 0 ? I 20:24:04 00:00:00 [kworker/0:3-memcg_kmem_cache] ubuntu 1680 14878 0.0 0.0 2608 604 pts/0 Ss+ 20:24:13 00:00:00 /bin/sh -c /usr/bin/python3 /home/ubuntu/.ansible/tmp/ansible-tmp-1643747053.37-114999225594279/AnsiballZ_command.py && sleep 0 ubuntu 14878 14879 0.0 0.1 28120 17848 pts/0 S+ 20:24:13 00:00:00 /usr/bin/python3 /home/ubuntu/.ansible/tmp/ansible-tmp-1643747053.37-114999225594279/AnsiballZ_command.py ubuntu 14879 14881 0.0 0.0 2608 608 pts/0 S+ 20:24:13 00:00:00 /bin/sh -c # workaround get-pip breaking when setuptools installed # ref: https://github.com/pypa/pip/issues/10742 sudo -H -E pip3 uninstall setuptools -y || true ./unstack.sh > /dev/null 2>&1 ./stack.sh > /dev/null 2>&1 ubuntu 1598 14885 0.0 0.0 7108 4136 ? Ss 20:24:14 00:00:00 /usr/bin/dbus-daemon --session --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only root 2 16126 0.0 0.0 2488 584 ? S 20:24:17 00:00:00 bpfilter_umh root 2 17250 0.0 0.0 0 0 ? I 20:24:19 00:00:00 [kworker/2:0-events] ubuntu 14881 17622 0.5 0.0 15056 10316 pts/0 S+ 20:24:20 00:00:03 bash ./stack.sh ubuntu 17622 19881 0.0 0.0 10636 3816 pts/0 S+ 20:24:24 00:00:00 bash ./stack.sh ubuntu 17622 19886 0.0 0.0 10636 3816 pts/0 S+ 20:24:24 00:00:00 bash ./stack.sh ubuntu 19881 19888 0.1 0.0 17384 11380 pts/0 S+ 20:24:24 00:00:00 /usr/bin/python3.8 /home/ubuntu/devstack/tools/outfilter.py -v --no-timestamp -o /opt/stack/logs/stack.sh.log.2022-02-01-202424 ubuntu 19886 19893 0.0 0.0 17384 11180 pts/0 S+ 20:24:24 00:00:00 /usr/bin/python3.8 /home/ubuntu/devstack/tools/outfilter.py -o /opt/stack/logs/stack.sh.log.2022-02-01-202424.summary.2022-02-01-202424 root 1 21680 0.0 0.1 283808 15824 ? Ssl 20:24:26 00:00:00 /usr/lib/packagekit/packagekitd memcache 1 24701 0.0 0.0 409768 3912 ? Ssl 20:25:45 00:00:00 /usr/bin/memcached -m 64 -p 11211 -u memcache -l 127.0.0.1 -P /var/run/memcached/memcached.pid root 1 24931 0.0 0.0 7744 2932 ? Ss 20:25:49 00:00:00 /usr/sbin/conntrackd -C /etc/conntrackd/conntrackd.conf root 1 25114 0.0 0.0 15304 8880 ? Ss 20:25:51 00:00:00 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock haproxy 25114 25115 0.0 0.0 236500 3104 ? Sl 20:25:51 00:00:00 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock root 1 26619 0.0 0.0 12176 6872 ? Ss 20:26:01 00:00:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups pcp 1 27727 0.3 0.0 8360 4452 ? Ss 20:26:12 00:00:02 /usr/lib/pcp/bin/pmcd root 27727 27730 0.0 0.0 7736 3836 ? S 20:26:12 00:00:00 /var/lib/pcp/pmdas/root/pmdaroot root 27730 27731 0.0 0.0 8308 5000 ? S 20:26:12 00:00:00 /var/lib/pcp/pmdas/proc/pmdaproc -d 3 root 27730 27732 0.0 0.0 7212 1688 ? S 20:26:12 00:00:00 /var/lib/pcp/pmdas/xfs/pmdaxfs -d 11 root 27730 27733 0.1 0.0 8244 5200 ? S 20:26:12 00:00:00 /var/lib/pcp/pmdas/linux/pmdalinux root 27730 27734 0.0 0.0 7264 3736 ? S 20:26:12 00:00:00 /var/lib/pcp/pmdas/kvm/pmdakvm -d 95 systemd+ 1 38929 0.0 0.0 34940 7804 ? Ss 20:26:28 00:00:00 /lib/systemd/systemd-networkd root 2 38950 0.0 0.0 0 0 ? I 20:26:28 00:00:00 [kworker/1:4-events] systemd+ 1 38953 0.0 0.0 23900 12136 ? Ss 20:26:28 00:00:00 /lib/systemd/systemd-resolved root 2 38965 0.0 0.0 0 0 ? I 20:26:28 00:00:00 [kworker/3:2-events] systemd+ 1 39081 0.0 0.0 90232 6124 ? Ssl 20:26:29 00:00:00 /lib/systemd/systemd-timesyncd syslog 1 43794 0.0 0.0 224348 4400 ? Ssl 20:26:42 00:00:00 /usr/sbin/rsyslogd -n -iNONE root 1 53195 0.0 0.0 35008 10568 ? S mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: mtu 1500 qdisc mq state UP group default qlen 1000 link/ether fa:16:3e:91:f1:e7 brd ff:ff:ff:ff:ff:ff inet 10.222.0.13/22 brd 10.222.3.255 scope global dynamic eth0 valid_lft 85832sec preferred_lft 85832sec inet6 fe80::f816:3eff:fe91:f1e7/64 scope link valid_lft forever preferred_lft forever 3: eth1: mtu 1500 qdisc mq state UP group default qlen 1000 link/ether fa:16:3e:0f:d5:05 brd ff:ff:ff:ff:ff:ff inet 192.168.0.47/22 brd 192.168.3.255 scope global dynamic eth1 valid_lft 85832sec preferred_lft 85832sec inet6 fe80::f816:3eff:fe0f:d505/64 scope link valid_lft forever preferred_lft forever ip route -------- default via 10.222.0.1 dev eth0 proto dhcp src 10.222.0.13 metric 100 10.222.0.0/22 dev eth0 proto kernel scope link src 10.222.0.13 10.222.0.0/22 dev eth1 proto dhcp scope link src 192.168.0.47 metric 100 169.254.169.254 via 192.168.0.12 dev eth1 proto dhcp src 192.168.0.47 metric 100 169.254.169.254 via 10.222.0.12 dev eth0 proto dhcp src 10.222.0.13 metric 100 192.168.0.0/22 dev eth1 proto kernel scope link src 192.168.0.47 192.168.0.0/22 dev eth0 proto dhcp scope link src 10.222.0.13 metric 100 ip -6 route ----------- ::1 dev lo proto kernel metric 256 pref medium fe80::/64 dev eth1 proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium ip netns -------- Open vSwitch Dump ================= sudo ovs-vsctl show ------------------- 4cc6949c-2400-4a39-add3-cc998723e793 ovs_version: "2.13.3" IP Tables Dump ============== sudo iptables --line-numbers -L -nv -t filter --------------------------------------------- Chain INPUT (policy ACCEPT 103K packets, 353M bytes) num pkts bytes target prot opt in out source destination Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) num pkts bytes target prot opt in out source destination Chain OUTPUT (policy ACCEPT 53486 packets, 3372K bytes) num pkts bytes target prot opt in out source destination sudo iptables --line-numbers -L -nv -t nat ------------------------------------------ Chain PREROUTING (policy ACCEPT 1117 packets, 244K bytes) num pkts bytes target prot opt in out source destination Chain INPUT (policy ACCEPT 739 packets, 219K bytes) num pkts bytes target prot opt in out source destination Chain OUTPUT (policy ACCEPT 534 packets, 42477 bytes) num pkts bytes target prot opt in out source destination Chain POSTROUTING (policy ACCEPT 534 packets, 42477 bytes) num pkts bytes target prot opt in out source destination sudo iptables --line-numbers -L -nv -t mangle --------------------------------------------- Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes) num pkts bytes target prot opt in out source destination Chain INPUT (policy ACCEPT 0 packets, 0 bytes) num pkts bytes target prot opt in out source destination Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) num pkts bytes target prot opt in out source destination Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes) num pkts bytes target prot opt in out source destination Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes) num pkts bytes target prot opt in out source destination EB Tables Dump ============== sudo ebtables -t filter -L -------------------------- Bridge table: filter Bridge chain: INPUT, entries: 0, policy: ACCEPT Bridge chain: FORWARD, entries: 0, policy: ACCEPT Bridge chain: OUTPUT, entries: 0, policy: ACCEPT sudo ebtables -t nat -L ----------------------- Bridge table: nat Bridge chain: PREROUTING, entries: 0, policy: ACCEPT Bridge chain: OUTPUT, entries: 0, policy: ACCEPT Bridge chain: POSTROUTING, entries: 0, policy: ACCEPT Compute consoles ================ nova-compute Guru Meditation Report =================================== Skipping as nova-compute does not appear to be running neutron-dhcp-agent Guru Meditation Report ========================================= Skipping as neutron-dhcp-agent does not appear to be running neutron-l3-agent Guru Meditation Report ======================================= Skipping as neutron-l3-agent does not appear to be running neutron-linuxbridge-agent Guru Meditation Report ================================================ Skipping as neutron-linuxbridge-agent does not appear to be running neutron-metadata-agent Guru Meditation Report ============================================= Skipping as neutron-metadata-agent does not appear to be running neutron-openvswitch-agent Guru Meditation Report ================================================ Skipping as neutron-openvswitch-agent does not appear to be running cinder-volume Guru Meditation Report ==================================== Skipping as cinder-volume does not appear to be running