File System Summary =================== WARN: Device /dev/loop0 (/snap/core18/1997) is 100% full, might be an issue WARN: Device /dev/loop2 (/snap/snapd/11588) is 100% full, might be an issue WARN: Device /dev/loop1 (/snap/lxd/19647) is 100% full, might be an issue Filesystem Size Used Avail Use% Mounted on udev 5.9G 0 5.9G 0% /dev tmpfs 1.2G 1.3M 1.2G 1% /run /dev/sda1 39G 6.6G 33G 18% / tmpfs 5.9G 4.0K 5.9G 1% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 5.9G 0 5.9G 0% /sys/fs/cgroup /dev/loop0 56M 56M 0 100% /snap/core18/1997 /dev/loop2 33M 33M 0 100% /snap/snapd/11588 /dev/loop1 71M 71M 0 100% /snap/lxd/19647 /dev/sda15 105M 7.8M 97M 8% /boot/efi tmpfs 1.2G 0 1.2G 0% /run/user/1000 tmpfs 512M 123M 390M 24% /opt/stack/data/etcd /dev/loop3 6.0G 689M 5.4G 12% /opt/stack/data/swift/drives/sdb1 Process Listing =============== ps axo user,ppid,pid,pcpu,pmem,vsz,rss,tty,stat,start,time,args --------------------------------------------------------------- USER PPID PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND root 0 1 1.6 0.1 171100 13376 ? Ss 07:42:36 00:00:52 /lib/systemd/systemd --system --deserialize 27 root 0 2 0.0 0.0 0 0 ? S 07:42:36 00:00:00 [kthreadd] root 2 3 0.0 0.0 0 0 ? I< 07:42:36 00:00:00 [rcu_gp] root 2 4 0.0 0.0 0 0 ? I< 07:42:36 00:00:00 [rcu_par_gp] root 2 6 0.0 0.0 0 0 ? I< 07:42:36 00:00:00 [kworker/0:0H-kblockd] root 2 8 0.0 0.0 0 0 ? I< 07:42:36 00:00:00 [mm_percpu_wq] root 2 9 0.0 0.0 0 0 ? S 07:42:36 00:00:00 [ksoftirqd/0] root 2 10 0.0 0.0 0 0 ? I 07:42:36 00:00:02 [rcu_sched] root 2 11 0.0 0.0 0 0 ? S 07:42:36 00:00:00 [migration/0] root 2 12 0.0 0.0 0 0 ? S 07:42:36 00:00:00 [idle_inject/0] root 2 14 0.0 0.0 0 0 ? S 07:42:36 00:00:00 [cpuhp/0] root 2 15 0.0 0.0 0 0 ? S 07:42:36 00:00:00 [cpuhp/1] root 2 16 0.0 0.0 0 0 ? S 07:42:36 00:00:00 [idle_inject/1] root 2 17 0.0 0.0 0 0 ? S 07:42:36 00:00:00 [migration/1] root 2 18 0.0 0.0 0 0 ? S 07:42:36 00:00:00 [ksoftirqd/1] root 2 20 0.0 0.0 0 0 ? I< 07:42:36 00:00:00 [kworker/1:0H-kblockd] root 2 21 0.0 0.0 0 0 ? S 07:42:36 00:00:00 [cpuhp/2] root 2 22 0.0 0.0 0 0 ? S 07:42:36 00:00:00 [idle_inject/2] root 2 23 0.0 0.0 0 0 ? S 07:42:36 00:00:00 [migration/2] root 2 24 0.0 0.0 0 0 ? S 07:42:36 00:00:00 [ksoftirqd/2] root 2 26 0.0 0.0 0 0 ? I< 07:42:36 00:00:00 [kworker/2:0H-kblockd] root 2 27 0.0 0.0 0 0 ? S 07:42:36 00:00:00 [cpuhp/3] root 2 28 0.0 0.0 0 0 ? S 07:42:36 00:00:00 [idle_inject/3] root 2 29 0.0 0.0 0 0 ? S 07:42:36 00:00:00 [migration/3] root 2 30 0.0 0.0 0 0 ? S 07:42:36 00:00:00 [ksoftirqd/3] root 2 32 0.0 0.0 0 0 ? I< 07:42:36 00:00:00 [kworker/3:0H-kblockd] root 2 33 0.0 0.0 0 0 ? S 07:42:36 00:00:00 [kdevtmpfs] root 2 34 0.0 0.0 0 0 ? I< 07:42:36 00:00:00 [netns] root 2 35 0.0 0.0 0 0 ? S 07:42:36 00:00:00 [rcu_tasks_kthre] root 2 36 0.0 0.0 0 0 ? S 07:42:36 00:00:00 [kauditd] root 2 40 0.0 0.0 0 0 ? S 07:42:36 00:00:00 [khungtaskd] root 2 41 0.0 0.0 0 0 ? S 07:42:36 00:00:00 [oom_reaper] root 2 42 0.0 0.0 0 0 ? I< 07:42:36 00:00:00 [writeback] root 2 43 0.0 0.0 0 0 ? S 07:42:36 00:00:00 [kcompactd0] root 2 44 0.0 0.0 0 0 ? SN 07:42:36 00:00:00 [ksmd] root 2 45 0.0 0.0 0 0 ? SN 07:42:36 00:00:00 [khugepaged] root 2 91 0.0 0.0 0 0 ? I< 07:42:36 00:00:00 [kintegrityd] root 2 92 0.0 0.0 0 0 ? I< 07:42:36 00:00:00 [kblockd] root 2 93 0.0 0.0 0 0 ? I< 07:42:36 00:00:00 [blkcg_punt_bio] root 2 94 0.0 0.0 0 0 ? I< 07:42:36 00:00:00 [tpm_dev_wq] root 2 95 0.0 0.0 0 0 ? I< 07:42:36 00:00:00 [ata_sff] root 2 96 0.0 0.0 0 0 ? I< 07:42:36 00:00:00 [md] root 2 97 0.0 0.0 0 0 ? I< 07:42:36 00:00:00 [edac-poller] root 2 98 0.0 0.0 0 0 ? I< 07:42:36 00:00:00 [devfreq_wq] root 2 99 0.0 0.0 0 0 ? S 07:42:37 00:00:00 [watchdogd] root 2 104 0.0 0.0 0 0 ? S 07:42:37 00:00:00 [kswapd0] root 2 105 0.0 0.0 0 0 ? S 07:42:37 00:00:00 [ecryptfs-kthrea] root 2 107 0.0 0.0 0 0 ? I< 07:42:37 00:00:00 [kthrotld] root 2 108 0.0 0.0 0 0 ? I< 07:42:37 00:00:00 [acpi_thermal_pm] root 2 110 0.0 0.0 0 0 ? S 07:42:37 00:00:00 [scsi_eh_0] root 2 111 0.0 0.0 0 0 ? I< 07:42:37 00:00:00 [scsi_tmf_0] root 2 112 0.0 0.0 0 0 ? S 07:42:37 00:00:00 [scsi_eh_1] root 2 113 0.0 0.0 0 0 ? I< 07:42:37 00:00:00 [scsi_tmf_1] root 2 114 0.0 0.0 0 0 ? I 07:42:37 00:00:01 [kworker/u256:2-events_power_efficient] root 2 115 0.0 0.0 0 0 ? I< 07:42:37 00:00:00 [vfio-irqfd-clea] root 2 116 0.0 0.0 0 0 ? I< 07:42:38 00:00:00 [ipv6_addrconf] root 2 126 0.0 0.0 0 0 ? I< 07:42:38 00:00:00 [kstrp] root 2 129 0.0 0.0 0 0 ? I< 07:42:38 00:00:00 [kworker/u257:0] root 2 142 0.0 0.0 0 0 ? I< 07:42:38 00:00:00 [charger_manager] root 2 184 0.0 0.0 0 0 ? I< 07:42:39 00:00:00 [hv_vmbus_con] root 2 185 0.0 0.0 0 0 ? I< 07:42:39 00:00:00 [hv_pri_chan] root 2 186 0.0 0.0 0 0 ? I< 07:42:39 00:00:00 [hv_sub_chan] root 2 187 0.0 0.0 0 0 ? I< 07:42:39 00:00:00 [kworker/3:1H-kblockd] root 2 188 0.0 0.0 0 0 ? I< 07:42:39 00:00:00 [cryptd] root 2 216 0.0 0.0 0 0 ? S 07:42:39 00:00:00 [scsi_eh_2] root 2 217 0.0 0.0 0 0 ? S 07:42:39 00:00:00 [scsi_eh_3] root 2 219 0.0 0.0 0 0 ? I< 07:42:39 00:00:00 [scsi_tmf_3] root 2 220 0.0 0.0 0 0 ? I< 07:42:39 00:00:00 [scsi_tmf_2] root 2 221 0.0 0.0 0 0 ? I< 07:42:39 00:00:00 [storvsc_error_w] root 2 233 0.0 0.0 0 0 ? I< 07:42:39 00:00:00 [storvsc_error_w] root 2 236 0.0 0.0 0 0 ? I< 07:42:39 00:00:00 [kworker/0:1H-kblockd] root 2 239 0.0 0.0 0 0 ? I< 07:42:39 00:00:00 [kworker/1:1H-kblockd] root 2 274 0.0 0.0 0 0 ? I< 07:42:41 00:00:00 [raid5wq] root 2 314 0.1 0.0 0 0 ? S 07:42:41 00:00:03 [jbd2/sda1-8] root 2 315 0.0 0.0 0 0 ? I< 07:42:41 00:00:00 [ext4-rsv-conver] root 2 316 0.0 0.0 0 0 ? I< 07:42:41 00:00:00 [kworker/2:1H-kblockd] root 2 413 0.0 0.0 0 0 ? I 07:42:43 00:00:00 [kworker/0:4-cgroup_destroy] root 1 421 0.0 0.0 19360 5656 ? Ss 07:42:44 00:00:01 /lib/systemd/systemd-udevd root 2 448 0.0 0.0 0 0 ? S 07:42:44 00:00:00 [hv_balloon] root 2 549 0.0 0.0 0 0 ? I< 07:42:45 00:00:00 [kaluad] root 2 550 0.0 0.0 0 0 ? I< 07:42:45 00:00:00 [kmpath_rdacd] root 2 551 0.0 0.0 0 0 ? I< 07:42:45 00:00:00 [kmpathd] root 2 552 0.0 0.0 0 0 ? I< 07:42:45 00:00:00 [kmpath_handlerd] root 1 553 0.0 0.1 280136 17940 ? SLsl 07:42:45 00:00:00 /sbin/multipathd -d -s root 2 562 0.0 0.0 0 0 ? S< 07:42:45 00:00:00 [loop0] root 2 567 0.0 0.0 0 0 ? S< 07:42:45 00:00:00 [loop1] root 2 569 0.0 0.0 0 0 ? S< 07:42:45 00:00:00 [loop2] root 1 765 0.0 0.0 241048 9416 ? Ssl 07:42:58 00:00:00 /usr/lib/accountsservice/accounts-daemon root 1 771 0.0 0.0 8536 2804 ? Ss 07:42:58 00:00:00 /usr/sbin/cron -f message+ 1 773 0.1 0.0 7780 4692 ? Ss 07:42:58 00:00:05 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only root 1 786 0.0 0.0 81828 3740 ? Ssl 07:42:58 00:00:00 /usr/sbin/irqbalance --foreground root 1 787 0.0 0.1 29264 17872 ? Ss 07:42:58 00:00:00 /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers root 1 791 0.1 0.3 1226068 38584 ? Ssl 07:42:58 00:00:03 /usr/lib/snapd/snapd root 1 792 0.0 0.0 16780 7556 ? Ss 07:42:58 00:00:02 /lib/systemd/systemd-logind daemon 1 794 0.0 0.0 3792 2260 ? Ss 07:42:58 00:00:00 /usr/sbin/atd -f root 1 818 0.0 0.0 7352 2172 ttyS0 Ss+ 07:42:58 00:00:00 /sbin/agetty -o -p -- \u --keep-baud 115200,38400,9600 ttyS0 vt220 root 1 827 0.0 0.0 5828 1716 tty1 Ss+ 07:42:58 00:00:00 /sbin/agetty -o -p -- \u --noclear tty1 linux root 1 846 0.0 0.1 108100 19832 ? Ssl 07:42:58 00:00:00 /usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal root 1 858 0.0 0.0 237748 10108 ? Ssl 07:42:59 00:00:00 /usr/lib/policykit-1/polkitd --no-debug root 1 1555 0.0 0.0 13796 6684 ? Ss 07:48:05 00:00:00 sshd: ubuntu [priv] ubuntu 1 1572 0.0 0.0 18496 9800 ? Ss 07:48:05 00:00:00 /lib/systemd/systemd --user ubuntu 1572 1573 0.0 0.0 104240 4524 ? S 07:48:05 00:00:00 (sd-pam) ubuntu 1555 1649 0.0 0.0 13928 5224 ? S 07:48:06 00:00:00 sshd: ubuntu@pts/0 ubuntu 1649 14975 0.0 0.0 2608 528 pts/0 Ss+ 08:01:03 00:00:00 /bin/sh -c /usr/bin/python3 /home/ubuntu/.ansible/tmp/ansible-tmp-1647417664.45-112870032416313/AnsiballZ_command.py && sleep 0 ubuntu 14975 14976 0.0 0.1 28084 17844 pts/0 S+ 08:01:03 00:00:00 /usr/bin/python3 /home/ubuntu/.ansible/tmp/ansible-tmp-1647417664.45-112870032416313/AnsiballZ_command.py ubuntu 14976 14978 0.0 0.0 2608 592 pts/0 S+ 08:01:04 00:00:00 /bin/sh -c # workaround get-pip breaking when setuptools installed # ref: https://github.com/pypa/pip/issues/10742 sudo -H -E pip3 uninstall setuptools -y || true ./unstack.sh > /dev/null 2>&1 ./stack.sh > /dev/null 2>&1 ubuntu 1572 14982 0.0 0.0 7348 4344 ? Ss 08:01:04 00:00:00 /usr/bin/dbus-daemon --session --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only root 2 16231 0.0 0.0 2488 508 ? S 08:01:07 00:00:00 bpfilter_umh root 2 17362 0.0 0.0 0 0 ? I 08:01:09 00:00:00 [kworker/1:3-events] ubuntu 14978 17737 0.4 0.0 15760 10932 pts/0 S+ 08:01:11 00:00:08 bash ./stack.sh ubuntu 17737 19999 0.0 0.0 10648 3164 pts/0 S+ 08:01:14 00:00:00 bash ./stack.sh ubuntu 17737 20004 0.0 0.0 10648 3164 pts/0 S+ 08:01:14 00:00:00 bash ./stack.sh ubuntu 19999 20008 0.1 0.0 17516 11232 pts/0 S+ 08:01:14 00:00:02 /usr/bin/python3.8 /home/ubuntu/devstack/tools/outfilter.py -v --no-timestamp -o /opt/stack/logs/stack.sh.log.2022-03-16-080115 ubuntu 20004 20011 0.0 0.0 17384 11292 pts/0 S+ 08:01:14 00:00:00 /usr/bin/python3.8 /home/ubuntu/devstack/tools/outfilter.py -o /opt/stack/logs/stack.sh.log.2022-03-16-080115.summary.2022-03-16-080115 root 2 24977 0.0 0.0 0 0 ? I 08:04:29 00:00:00 [kworker/2:0-rcu_par_gp] uuidd 1 25039 0.0 0.0 9744 1064 ? Ss 08:04:33 00:00:00 /usr/sbin/uuidd --socket-activation root 1 25122 0.0 0.0 8136 3728 ? Ss 08:04:38 00:00:00 /usr/sbin/conntrackd -C /etc/conntrackd/conntrackd.conf root 1 25369 0.0 0.0 15300 9072 ? Ss 08:04:48 00:00:00 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock haproxy 25369 25380 0.0 0.0 236496 2892 ? Sl 08:04:48 00:00:00 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock root 1 26249 0.0 0.0 12172 6640 ? Ss 08:05:09 00:00:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups pcp 1 28174 0.1 0.0 8388 4432 ? Ss 08:05:39 00:00:02 /usr/lib/pcp/bin/pmcd root 28174 28177 0.0 0.0 7736 3904 ? S 08:05:39 00:00:00 /var/lib/pcp/pmdas/root/pmdaroot root 28177 28178 0.0 0.0 8440 5008 ? S 08:05:39 00:00:00 /var/lib/pcp/pmdas/proc/pmdaproc -d 3 root 28177 28179 0.0 0.0 7212 1676 ? S 08:05:39 00:00:00 /var/lib/pcp/pmdas/xfs/pmdaxfs -d 11 root 28177 28180 0.3 0.0 8268 5032 ? S 08:05:39 00:00:06 /var/lib/pcp/pmdas/linux/pmdalinux root 28177 28181 0.0 0.0 7268 3800 ? S 08:05:39 00:00:00 /var/lib/pcp/pmdas/kvm/pmdakvm -d 95 root 2 34527 0.0 0.0 0 0 ? I 08:05:49 00:00:00 [kworker/3:3-rcu_par_gp] systemd+ 1 39391 0.0 0.0 26740 7856 ? Ss 08:05:56 00:00:00 /lib/systemd/systemd-networkd systemd+ 1 39408 0.0 0.0 23896 10384 ? Ss 08:05:56 00:00:00 /lib/systemd/systemd-resolved systemd+ 1 39535 0.0 0.0 90224 6136 ? Ssl 08:05:58 00:00:00 /lib/systemd/systemd-timesyncd syslog 1 44248 0.1 0.0 224344 4160 ? Ssl 08:06:14 00:00:02 /usr/sbin/rsyslogd -n -iNONE root 1 53708 0.4 0.5 114976 67092 ? S mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: mtu 1500 qdisc mq state UP group default qlen 1000 link/ether fa:16:3e:fb:87:7b brd ff:ff:ff:ff:ff:ff inet 10.222.0.20/22 brd 10.222.3.255 scope global dynamic eth0 valid_lft 84698sec preferred_lft 84698sec inet6 fe80::f816:3eff:fefb:877b/64 scope link valid_lft forever preferred_lft forever 3: eth1: mtu 1500 qdisc mq master ovs-system state UP group default qlen 1000 link/ether fa:16:3e:01:38:af brd ff:ff:ff:ff:ff:ff inet6 fe80::f816:3eff:fe01:38af/64 scope link valid_lft forever preferred_lft forever 4: ovs-system: mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether e2:9f:74:ea:fb:7f brd ff:ff:ff:ff:ff:ff 5: br-int: mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 42:ca:6f:f0:9e:41 brd ff:ff:ff:ff:ff:ff 6: br-data: mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether ce:0f:16:8c:13:47 brd ff:ff:ff:ff:ff:ff inet 192.168.0.29/22 brd 192.168.3.255 scope global br-data valid_lft forever preferred_lft forever inet 172.25.5.1/24 scope global br-data valid_lft forever preferred_lft forever inet6 fe80::cc0f:16ff:fe8c:1347/64 scope link valid_lft forever preferred_lft forever ip route -------- default via 10.222.0.1 dev eth0 proto dhcp src 10.222.0.20 metric 100 10.222.0.0/22 dev eth0 proto kernel scope link src 10.222.0.20 169.254.169.254 via 10.222.0.12 dev eth0 proto dhcp src 10.222.0.20 metric 100 172.25.5.0/24 dev br-data proto kernel scope link src 172.25.5.1 192.168.0.0/22 dev br-data proto kernel scope link src 192.168.0.29 192.168.0.0/22 dev eth0 proto dhcp scope link src 10.222.0.20 metric 100 ip -6 route ----------- ::1 dev lo proto kernel metric 256 pref medium fe80::/64 dev eth1 proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium fe80::/64 dev br-data proto kernel metric 256 pref medium ip netns -------- qdhcp-97ff25d7-fa21-452c-bd19-9f37b0e843e9 (id: 2) qrouter-0b397830-6076-48fc-ac7c-6c3664b74663 (id: 1) qdhcp-2751ab34-edf2-49ae-a8e7-d3f44f4268fb (id: 0) sudo ip netns exec qdhcp-97ff25d7-fa21-452c-bd19-9f37b0e843e9 ip neigh ---------------------------------------------------------------------- sudo ip netns exec qdhcp-97ff25d7-fa21-452c-bd19-9f37b0e843e9 ip addr --------------------------------------------------------------------- 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 10: tap8400fca1-66: mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether fa:16:3e:ce:45:fc brd ff:ff:ff:ff:ff:ff inet 192.168.233.2/24 brd 192.168.233.255 scope global tap8400fca1-66 valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fece:45fc/64 scope link valid_lft forever preferred_lft forever sudo ip netns exec qdhcp-97ff25d7-fa21-452c-bd19-9f37b0e843e9 ip route ---------------------------------------------------------------------- default via 192.168.233.1 dev tap8400fca1-66 proto static 192.168.233.0/24 dev tap8400fca1-66 proto kernel scope link src 192.168.233.2 sudo ip netns exec qdhcp-97ff25d7-fa21-452c-bd19-9f37b0e843e9 ip -6 route ------------------------------------------------------------------------- fe80::/64 dev tap8400fca1-66 proto kernel metric 256 pref medium sudo ip netns exec qrouter-0b397830-6076-48fc-ac7c-6c3664b74663 ip neigh ------------------------------------------------------------------------ fe80::cc0f:16ff:fe8c:1347 dev qg-de87af78-68 lladdr ce:0f:16:8c:13:47 STALE fe80::f816:3eff:fe72:247e dev qg-de87af78-68 lladdr fa:16:3e:72:24:7e STALE sudo ip netns exec qrouter-0b397830-6076-48fc-ac7c-6c3664b74663 ip addr ----------------------------------------------------------------------- 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 8: qr-1582a184-26: mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether fa:16:3e:33:60:f7 brd ff:ff:ff:ff:ff:ff inet 172.20.0.1/26 brd 172.20.0.63 scope global qr-1582a184-26 valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe33:60f7/64 scope link valid_lft forever preferred_lft forever 9: qg-de87af78-68: mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether fa:16:3e:a9:28:77 brd ff:ff:ff:ff:ff:ff inet 172.25.5.162/24 brd 172.25.5.255 scope global qg-de87af78-68 valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fea9:2877/64 scope link valid_lft forever preferred_lft forever sudo ip netns exec qrouter-0b397830-6076-48fc-ac7c-6c3664b74663 ip route ------------------------------------------------------------------------ default via 172.25.5.1 dev qg-de87af78-68 proto static 172.20.0.0/26 dev qr-1582a184-26 proto kernel scope link src 172.20.0.1 172.25.5.0/24 dev qg-de87af78-68 proto kernel scope link src 172.25.5.162 sudo ip netns exec qrouter-0b397830-6076-48fc-ac7c-6c3664b74663 ip -6 route --------------------------------------------------------------------------- fe80::/64 dev qr-1582a184-26 proto kernel metric 256 pref medium fe80::/64 dev qg-de87af78-68 proto kernel metric 256 pref medium sudo ip netns exec qdhcp-2751ab34-edf2-49ae-a8e7-d3f44f4268fb ip neigh ---------------------------------------------------------------------- sudo ip netns exec qdhcp-2751ab34-edf2-49ae-a8e7-d3f44f4268fb ip addr --------------------------------------------------------------------- 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 7: tapf8eaea93-ff: mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether fa:16:3e:da:8a:41 brd ff:ff:ff:ff:ff:ff inet 172.20.0.2/26 brd 172.20.0.63 scope global tapf8eaea93-ff valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:feda:8a41/64 scope link valid_lft forever preferred_lft forever sudo ip netns exec qdhcp-2751ab34-edf2-49ae-a8e7-d3f44f4268fb ip route ---------------------------------------------------------------------- default via 172.20.0.1 dev tapf8eaea93-ff proto static 172.20.0.0/26 dev tapf8eaea93-ff proto kernel scope link src 172.20.0.2 sudo ip netns exec qdhcp-2751ab34-edf2-49ae-a8e7-d3f44f4268fb ip -6 route ------------------------------------------------------------------------- fe80::/64 dev tapf8eaea93-ff proto kernel metric 256 pref medium Open vSwitch Dump ================= sudo ovs-vsctl show ------------------- 32b17826-8f55-4b63-b071-9b6937291dff Manager "ptcp:6640:127.0.0.1" is_connected: true Bridge br-data Controller "tcp:127.0.0.1:6633" is_connected: true fail_mode: secure datapath_type: system Port eth1 Interface eth1 Port br-data Interface br-data type: internal Port phy-br-data Interface phy-br-data type: patch options: {peer=int-br-data} Bridge br-int Controller "tcp:127.0.0.1:6633" is_connected: true fail_mode: secure datapath_type: system Port br-int Interface br-int type: internal Port qg-de87af78-68 tag: 2 Interface qg-de87af78-68 type: internal Port tapf8eaea93-ff tag: 1 Interface tapf8eaea93-ff type: internal Port qr-1582a184-26 tag: 1 Interface qr-1582a184-26 type: internal Port tap8400fca1-66 tag: 3 Interface tap8400fca1-66 type: internal Port int-br-data Interface int-br-data type: patch options: {peer=phy-br-data} ovs_version: "2.13.5" sudo ovs-ofctl --protocols=OpenFlow10,OpenFlow11,OpenFlow12,OpenFlow13,OpenFlow14,OpenFlow15 show br-data --------------------------------------------------------------------------------------------------------- OFPT_FEATURES_REPLY (OF1.4) (xid=0x2): dpid:0000ce0f168c1347 n_tables:254, n_buffers:0 capabilities: FLOW_STATS TABLE_STATS PORT_STATS GROUP_STATS QUEUE_STATS BUNDLES OFPST_PORT_DESC reply (OF1.4) (xid=0x3): 1(eth1): addr:fa:16:3e:01:38:af config: 0 state: LIVE speed: 0 Mbps now, 0 Mbps max 2(phy-br-data): addr:de:79:c4:87:c3:49 config: 0 state: LIVE speed: 0 Mbps now, 0 Mbps max LOCAL(br-data): addr:ce:0f:16:8c:13:47 config: 0 state: LIVE speed: 0 Mbps now, 0 Mbps max OFPT_GET_CONFIG_REPLY (OF1.4) (xid=0x5): frags=normal miss_send_len=0 sudo ovs-ofctl --protocols=OpenFlow10,OpenFlow11,OpenFlow12,OpenFlow13,OpenFlow14,OpenFlow15 show br-int -------------------------------------------------------------------------------------------------------- OFPT_FEATURES_REPLY (OF1.4) (xid=0x2): dpid:000042ca6ff09e41 n_tables:254, n_buffers:0 capabilities: FLOW_STATS TABLE_STATS PORT_STATS GROUP_STATS QUEUE_STATS BUNDLES OFPST_PORT_DESC reply (OF1.4) (xid=0x3): 1(int-br-data): addr:a6:21:2a:c0:cb:96 config: 0 state: LIVE speed: 0 Mbps now, 0 Mbps max 2(tapf8eaea93-ff): addr:fa:16:3e:da:8a:41 config: 0 state: LIVE speed: 0 Mbps now, 0 Mbps max 3(qr-1582a184-26): addr:fa:16:3e:33:60:f7 config: 0 state: LIVE speed: 0 Mbps now, 0 Mbps max 4(qg-de87af78-68): addr:fa:16:3e:a9:28:77 config: 0 state: LIVE speed: 0 Mbps now, 0 Mbps max 5(tap8400fca1-66): addr:fa:16:3e:ce:45:fc config: 0 state: LIVE speed: 0 Mbps now, 0 Mbps max LOCAL(br-int): addr:42:ca:6f:f0:9e:41 config: PORT_DOWN state: LINK_DOWN speed: 0 Mbps now, 0 Mbps max OFPT_GET_CONFIG_REPLY (OF1.4) (xid=0x5): frags=normal miss_send_len=0 sudo ovs-ofctl --protocols=OpenFlow10,OpenFlow11,OpenFlow12,OpenFlow13,OpenFlow14,OpenFlow15 dump-ports-desc br-data -------------------------------------------------------------------------------------------------------------------- OFPST_PORT_DESC reply (OF1.4) (xid=0x2): 1(eth1): addr:fa:16:3e:01:38:af config: 0 state: LIVE speed: 0 Mbps now, 0 Mbps max 2(phy-br-data): addr:de:79:c4:87:c3:49 config: 0 state: LIVE speed: 0 Mbps now, 0 Mbps max LOCAL(br-data): addr:ce:0f:16:8c:13:47 config: 0 state: LIVE speed: 0 Mbps now, 0 Mbps max sudo ovs-ofctl --protocols=OpenFlow10,OpenFlow11,OpenFlow12,OpenFlow13,OpenFlow14,OpenFlow15 dump-ports-desc br-int ------------------------------------------------------------------------------------------------------------------- OFPST_PORT_DESC reply (OF1.4) (xid=0x2): 1(int-br-data): addr:a6:21:2a:c0:cb:96 config: 0 state: LIVE speed: 0 Mbps now, 0 Mbps max 2(tapf8eaea93-ff): addr:fa:16:3e:da:8a:41 config: 0 state: LIVE speed: 0 Mbps now, 0 Mbps max 3(qr-1582a184-26): addr:fa:16:3e:33:60:f7 config: 0 state: LIVE speed: 0 Mbps now, 0 Mbps max 4(qg-de87af78-68): addr:fa:16:3e:a9:28:77 config: 0 state: LIVE speed: 0 Mbps now, 0 Mbps max 5(tap8400fca1-66): addr:fa:16:3e:ce:45:fc config: 0 state: LIVE speed: 0 Mbps now, 0 Mbps max LOCAL(br-int): addr:42:ca:6f:f0:9e:41 config: PORT_DOWN state: LINK_DOWN speed: 0 Mbps now, 0 Mbps max sudo ovs-ofctl --protocols=OpenFlow10,OpenFlow11,OpenFlow12,OpenFlow13,OpenFlow14,OpenFlow15 dump-ports br-data --------------------------------------------------------------------------------------------------------------- OFPST_PORT reply (OF1.4) (xid=0x2): 3 ports port LOCAL: rx pkts=820, bytes=133674, drop=0, errs=0, frame=0, over=0, crc=0 tx pkts=13, bytes=1006, drop=0, errs=0, coll=0 duration=765.081s port 1: rx pkts=3900, bytes=648085, drop=0, errs=0, frame=0, over=0, crc=0 tx pkts=66, bytes=5900, drop=0, errs=0, coll=0 duration=760.828s port 2: rx pkts=23, bytes=1642, drop=?, errs=?, frame=?, over=?, crc=? tx pkts=168, bytes=44687, drop=?, errs=?, coll=? duration=303.544s sudo ovs-ofctl --protocols=OpenFlow10,OpenFlow11,OpenFlow12,OpenFlow13,OpenFlow14,OpenFlow15 dump-ports br-int -------------------------------------------------------------------------------------------------------------- OFPST_PORT reply (OF1.4) (xid=0x2): 6 ports port 4: rx pkts=132, bytes=34665, drop=0, errs=0, frame=0, over=0, crc=0 tx pkts=20, bytes=1488, drop=0, errs=0, coll=0 duration=253.801s port LOCAL: rx pkts=0, bytes=0, drop=155, errs=0, frame=0, over=0, crc=0 tx pkts=0, bytes=0, drop=0, errs=0, coll=0 duration=765.314s port 1: rx pkts=168, bytes=44687, drop=?, errs=?, frame=?, over=?, crc=? tx pkts=23, bytes=1642, drop=?, errs=?, coll=? duration=303.581s port 5: rx pkts=0, bytes=0, drop=0, errs=0, frame=0, over=0, crc=0 tx pkts=5, bytes=446, drop=0, errs=0, coll=0 duration=106.316s port 2: rx pkts=6, bytes=304, drop=0, errs=0, frame=0, over=0, crc=0 tx pkts=5, bytes=446, drop=0, errs=0, coll=0 duration=275.874s port 3: rx pkts=0, bytes=0, drop=0, errs=0, frame=0, over=0, crc=0 tx pkts=13, bytes=998, drop=0, errs=0, coll=0 duration=257.824s sudo ovs-ofctl --protocols=OpenFlow10,OpenFlow11,OpenFlow12,OpenFlow13,OpenFlow14,OpenFlow15 dump-flows br-data --------------------------------------------------------------------------------------------------------------- OFPST_FLOW reply (OF1.4) (xid=0x2): cookie=0xc2fa399105f0c66, duration=275.853s, table=0, n_packets=10, n_bytes=744, priority=4,in_port=2,dl_vlan=1 actions=set_field:4283->vlan_vid,NORMAL cookie=0xc2fa399105f0c66, duration=251.827s, table=0, n_packets=11, n_bytes=718, priority=4,in_port=2,dl_vlan=2 actions=pop_vlan,NORMAL cookie=0xc2fa399105f0c66, duration=105.715s, table=0, n_packets=2, n_bytes=180, priority=4,in_port=2,dl_vlan=3 actions=set_field:4294->vlan_vid,NORMAL cookie=0xc2fa399105f0c66, duration=303.550s, table=0, n_packets=0, n_bytes=0, priority=2,in_port=2 actions=drop cookie=0xc2fa399105f0c66, duration=303.612s, table=0, n_packets=168, n_bytes=44687, priority=0 actions=NORMAL sudo ovs-ofctl --protocols=OpenFlow10,OpenFlow11,OpenFlow12,OpenFlow13,OpenFlow14,OpenFlow15 dump-flows br-int -------------------------------------------------------------------------------------------------------------- OFPST_FLOW reply (OF1.4) (xid=0x2): cookie=0xc15618c838060179, duration=304.800s, table=0, n_packets=0, n_bytes=0, priority=65535,dl_vlan=4095 actions=drop cookie=0xc15618c838060179, duration=275.866s, table=0, n_packets=0, n_bytes=0, priority=3,in_port=1,dl_vlan=187 actions=set_field:4097->vlan_vid,goto_table:59 cookie=0xc15618c838060179, duration=251.837s, table=0, n_packets=132, n_bytes=36513, priority=3,in_port=1,vlan_tci=0x0000/0x1fff actions=push_vlan:0x8100,set_field:4098->vlan_vid,goto_table:59 cookie=0xc15618c838060179, duration=105.726s, table=0, n_packets=0, n_bytes=0, priority=3,in_port=1,dl_vlan=198 actions=set_field:4099->vlan_vid,goto_table:59 cookie=0xc15618c838060179, duration=303.576s, table=0, n_packets=36, n_bytes=8174, priority=2,in_port=1 actions=drop cookie=0xc15618c838060179, duration=304.832s, table=0, n_packets=43, n_bytes=3378, priority=0 actions=goto_table:59 cookie=0xc15618c838060179, duration=304.836s, table=23, n_packets=0, n_bytes=0, priority=0 actions=drop cookie=0xc15618c838060179, duration=304.803s, table=24, n_packets=0, n_bytes=0, priority=0 actions=drop cookie=0xc15618c838060179, duration=304.794s, table=30, n_packets=0, n_bytes=0, priority=0 actions=goto_table:59 cookie=0xc15618c838060179, duration=304.791s, table=31, n_packets=0, n_bytes=0, priority=0 actions=goto_table:59 cookie=0xc15618c838060179, duration=304.811s, table=59, n_packets=175, n_bytes=39891, priority=0 actions=goto_table:60 cookie=0xc15618c838060179, duration=273.827s, table=60, n_packets=5, n_bytes=446, priority=100,in_port=2 actions=set_field:0x2->reg5,set_field:0x1->reg6,resubmit(,73) cookie=0xc15618c838060179, duration=255.797s, table=60, n_packets=8, n_bytes=608, priority=100,in_port=3 actions=set_field:0x3->reg5,set_field:0x1->reg6,resubmit(,73) cookie=0xc15618c838060179, duration=249.735s, table=60, n_packets=11, n_bytes=718, priority=100,in_port=4 actions=set_field:0x4->reg5,set_field:0x2->reg6,resubmit(,73) cookie=0xc15618c838060179, duration=103.691s, table=60, n_packets=3, n_bytes=266, priority=100,in_port=5 actions=set_field:0x5->reg5,set_field:0x3->reg6,resubmit(,73) cookie=0xc15618c838060179, duration=304.808s, table=60, n_packets=148, n_bytes=37853, priority=3 actions=NORMAL cookie=0xc15618c838060179, duration=304.798s, table=62, n_packets=0, n_bytes=0, priority=3 actions=NORMAL cookie=0xc15618c838060179, duration=298.585s, table=71, n_packets=0, n_bytes=0, reset_counts priority=110,ct_state=+trk actions=ct_clear,resubmit(,71) cookie=0xc15618c838060179, duration=298.638s, table=71, n_packets=0, n_bytes=0, reset_counts priority=0 actions=drop cookie=0xc15618c838060179, duration=298.626s, table=72, n_packets=0, n_bytes=0, reset_counts priority=0 actions=drop cookie=0xc15618c838060179, duration=273.827s, table=73, n_packets=5, n_bytes=446, priority=80,reg5=0x2 actions=resubmit(,94) cookie=0xc15618c838060179, duration=255.797s, table=73, n_packets=8, n_bytes=608, priority=80,reg5=0x3 actions=resubmit(,94) cookie=0xc15618c838060179, duration=249.735s, table=73, n_packets=11, n_bytes=718, priority=80,reg5=0x4 actions=resubmit(,94) cookie=0xc15618c838060179, duration=103.691s, table=73, n_packets=3, n_bytes=266, priority=80,reg5=0x5 actions=resubmit(,94) cookie=0xc15618c838060179, duration=298.617s, table=73, n_packets=0, n_bytes=0, reset_counts priority=0 actions=drop cookie=0xc15618c838060179, duration=298.607s, table=81, n_packets=0, n_bytes=0, reset_counts priority=0 actions=drop cookie=0xc15618c838060179, duration=298.596s, table=82, n_packets=0, n_bytes=0, reset_counts priority=0 actions=drop cookie=0xc15618c838060179, duration=298.564s, table=91, n_packets=0, n_bytes=0, reset_counts priority=1 actions=resubmit(,94) cookie=0xc15618c838060179, duration=298.554s, table=92, n_packets=0, n_bytes=0, reset_counts priority=0 actions=drop cookie=0xc15618c838060179, duration=298.544s, table=93, n_packets=0, n_bytes=0, reset_counts priority=0 actions=drop cookie=0xc15618c838060179, duration=298.575s, table=94, n_packets=27, n_bytes=2038, reset_counts priority=1 actions=NORMAL IP Tables Dump ============== sudo iptables --line-numbers -L -nv -t filter --------------------------------------------- Chain INPUT (policy ACCEPT 737K packets, 3720M bytes) num pkts bytes target prot opt in out source destination Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) num pkts bytes target prot opt in out source destination Chain OUTPUT (policy ACCEPT 683K packets, 2704M bytes) num pkts bytes target prot opt in out source destination sudo iptables --line-numbers -L -nv -t nat ------------------------------------------ Chain PREROUTING (policy ACCEPT 110 packets, 34520 bytes) num pkts bytes target prot opt in out source destination Chain INPUT (policy ACCEPT 104 packets, 34112 bytes) num pkts bytes target prot opt in out source destination Chain OUTPUT (policy ACCEPT 57 packets, 3572 bytes) num pkts bytes target prot opt in out source destination Chain POSTROUTING (policy ACCEPT 57 packets, 3572 bytes) num pkts bytes target prot opt in out source destination 1 0 0 MASQUERADE all -- * eth0 172.25.5.0/24 0.0.0.0/0 sudo iptables --line-numbers -L -nv -t mangle --------------------------------------------- Chain PREROUTING (policy ACCEPT 259K packets, 3257M bytes) num pkts bytes target prot opt in out source destination Chain INPUT (policy ACCEPT 259K packets, 3257M bytes) num pkts bytes target prot opt in out source destination Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) num pkts bytes target prot opt in out source destination Chain OUTPUT (policy ACCEPT 257K packets, 2611M bytes) num pkts bytes target prot opt in out source destination Chain POSTROUTING (policy ACCEPT 257K packets, 2611M bytes) num pkts bytes target prot opt in out source destination EB Tables Dump ============== sudo ebtables -t filter -L -------------------------- Bridge table: filter Bridge chain: INPUT, entries: 0, policy: ACCEPT Bridge chain: FORWARD, entries: 0, policy: ACCEPT Bridge chain: OUTPUT, entries: 0, policy: ACCEPT sudo ebtables -t nat -L ----------------------- Bridge table: nat Bridge chain: PREROUTING, entries: 0, policy: ACCEPT Bridge chain: OUTPUT, entries: 0, policy: ACCEPT Bridge chain: POSTROUTING, entries: 0, policy: ACCEPT Compute consoles ================ nova-compute Guru Meditation Report =================================== Skipping as nova-compute does not appear to be running neutron-dhcp-agent Guru Meditation Report ========================================= 105756 killall -e -USR2 neutron-dhcp-agent ----------------------------------- guru meditation report in neutron-dhcp-agent log neutron-l3-agent Guru Meditation Report ======================================= 106131 killall -e -USR2 neutron-l3-agent --------------------------------- guru meditation report in neutron-l3-agent log neutron-linuxbridge-agent Guru Meditation Report ================================================ Skipping as neutron-linuxbridge-agent does not appear to be running neutron-metadata-agent Guru Meditation Report ============================================= 106486 106976 106977 killall -e -USR2 neutron-metadata-agent --------------------------------------- guru meditation report in neutron-metadata-agent log neutron-openvswitch-agent Guru Meditation Report ================================================ 105393 killall -e -USR2 neutron-openvswitch-agent ------------------------------------------ guru meditation report in neutron-openvswitch-agent log cinder-volume Guru Meditation Report ==================================== 114058 117304 killall -e -USR2 cinder-volume ------------------------------ guru meditation report in cinder-volume log