File System Summary =================== WARN: Device /dev/loop0 (/snap/core18/1997) is 100% full, might be an issue WARN: Device /dev/loop1 (/snap/lxd/19647) is 100% full, might be an issue WARN: Device /dev/loop2 (/snap/snapd/11588) is 100% full, might be an issue Filesystem Size Used Avail Use% Mounted on udev 5.9G 0 5.9G 0% /dev tmpfs 1.2G 1.3M 1.2G 1% /run /dev/sda1 39G 6.4G 33G 17% / tmpfs 5.9G 4.0K 5.9G 1% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 5.9G 0 5.9G 0% /sys/fs/cgroup /dev/loop0 56M 56M 0 100% /snap/core18/1997 /dev/loop1 71M 71M 0 100% /snap/lxd/19647 /dev/loop2 33M 33M 0 100% /snap/snapd/11588 /dev/sda15 105M 7.8M 97M 8% /boot/efi tmpfs 1.2G 0 1.2G 0% /run/user/1000 tmpfs 512M 123M 390M 24% /opt/stack/data/etcd /dev/loop3 6.0G 689M 5.4G 12% /opt/stack/data/swift/drives/sdb1 Process Listing =============== ps axo user,ppid,pid,pcpu,pmem,vsz,rss,tty,stat,start,time,args --------------------------------------------------------------- USER PPID PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND root 0 1 2.3 0.1 171104 13260 ? Ss 07:01:38 00:00:52 /lib/systemd/systemd --system --deserialize 27 root 0 2 0.0 0.0 0 0 ? S 07:01:38 00:00:00 [kthreadd] root 2 3 0.0 0.0 0 0 ? I< 07:01:38 00:00:00 [rcu_gp] root 2 4 0.0 0.0 0 0 ? I< 07:01:38 00:00:00 [rcu_par_gp] root 2 6 0.0 0.0 0 0 ? I< 07:01:38 00:00:00 [kworker/0:0H-kblockd] root 2 8 0.0 0.0 0 0 ? I< 07:01:38 00:00:00 [mm_percpu_wq] root 2 9 0.0 0.0 0 0 ? S 07:01:38 00:00:00 [ksoftirqd/0] root 2 10 0.0 0.0 0 0 ? I 07:01:38 00:00:02 [rcu_sched] root 2 11 0.0 0.0 0 0 ? S 07:01:38 00:00:00 [migration/0] root 2 12 0.0 0.0 0 0 ? S 07:01:38 00:00:00 [idle_inject/0] root 2 14 0.0 0.0 0 0 ? S 07:01:38 00:00:00 [cpuhp/0] root 2 15 0.0 0.0 0 0 ? S 07:01:38 00:00:00 [cpuhp/1] root 2 16 0.0 0.0 0 0 ? S 07:01:38 00:00:00 [idle_inject/1] root 2 17 0.0 0.0 0 0 ? S 07:01:38 00:00:00 [migration/1] root 2 18 0.0 0.0 0 0 ? S 07:01:38 00:00:00 [ksoftirqd/1] root 2 20 0.0 0.0 0 0 ? I< 07:01:38 00:00:00 [kworker/1:0H-kblockd] root 2 21 0.0 0.0 0 0 ? S 07:01:38 00:00:00 [cpuhp/2] root 2 22 0.0 0.0 0 0 ? S 07:01:38 00:00:00 [idle_inject/2] root 2 23 0.0 0.0 0 0 ? S 07:01:38 00:00:00 [migration/2] root 2 24 0.0 0.0 0 0 ? S 07:01:38 00:00:00 [ksoftirqd/2] root 2 26 0.0 0.0 0 0 ? I< 07:01:38 00:00:00 [kworker/2:0H-kblockd] root 2 27 0.0 0.0 0 0 ? S 07:01:38 00:00:00 [cpuhp/3] root 2 28 0.0 0.0 0 0 ? S 07:01:38 00:00:00 [idle_inject/3] root 2 29 0.0 0.0 0 0 ? S 07:01:38 00:00:00 [migration/3] root 2 30 0.0 0.0 0 0 ? S 07:01:38 00:00:00 [ksoftirqd/3] root 2 32 0.0 0.0 0 0 ? I< 07:01:38 00:00:00 [kworker/3:0H] root 2 33 0.0 0.0 0 0 ? S 07:01:38 00:00:00 [kdevtmpfs] root 2 34 0.0 0.0 0 0 ? I< 07:01:38 00:00:00 [netns] root 2 35 0.0 0.0 0 0 ? S 07:01:38 00:00:00 [rcu_tasks_kthre] root 2 36 0.0 0.0 0 0 ? S 07:01:38 00:00:00 [kauditd] root 2 40 0.0 0.0 0 0 ? S 07:01:38 00:00:00 [khungtaskd] root 2 41 0.0 0.0 0 0 ? S 07:01:38 00:00:00 [oom_reaper] root 2 42 0.0 0.0 0 0 ? I< 07:01:38 00:00:00 [writeback] root 2 43 0.0 0.0 0 0 ? S 07:01:38 00:00:00 [kcompactd0] root 2 44 0.0 0.0 0 0 ? SN 07:01:38 00:00:00 [ksmd] root 2 45 0.0 0.0 0 0 ? SN 07:01:38 00:00:00 [khugepaged] root 2 92 0.0 0.0 0 0 ? I< 07:01:38 00:00:00 [kintegrityd] root 2 93 0.0 0.0 0 0 ? I< 07:01:38 00:00:00 [kblockd] root 2 94 0.0 0.0 0 0 ? I< 07:01:38 00:00:00 [blkcg_punt_bio] root 2 95 0.0 0.0 0 0 ? I< 07:01:38 00:00:00 [tpm_dev_wq] root 2 96 0.0 0.0 0 0 ? I< 07:01:38 00:00:00 [ata_sff] root 2 97 0.0 0.0 0 0 ? I< 07:01:38 00:00:00 [md] root 2 98 0.0 0.0 0 0 ? I< 07:01:38 00:00:00 [edac-poller] root 2 99 0.0 0.0 0 0 ? I< 07:01:38 00:00:00 [devfreq_wq] root 2 100 0.0 0.0 0 0 ? S 07:01:39 00:00:00 [watchdogd] root 2 103 0.0 0.0 0 0 ? S 07:01:39 00:00:00 [kswapd0] root 2 104 0.0 0.0 0 0 ? S 07:01:39 00:00:00 [ecryptfs-kthrea] root 2 106 0.0 0.0 0 0 ? I< 07:01:39 00:00:00 [kthrotld] root 2 107 0.0 0.0 0 0 ? I< 07:01:39 00:00:00 [acpi_thermal_pm] root 2 111 0.0 0.0 0 0 ? S 07:01:39 00:00:00 [scsi_eh_0] root 2 112 0.0 0.0 0 0 ? I< 07:01:39 00:00:00 [scsi_tmf_0] root 2 113 0.0 0.0 0 0 ? S 07:01:39 00:00:00 [scsi_eh_1] root 2 114 0.0 0.0 0 0 ? I< 07:01:39 00:00:00 [scsi_tmf_1] root 2 116 0.0 0.0 0 0 ? I< 07:01:39 00:00:00 [vfio-irqfd-clea] root 2 117 0.0 0.0 0 0 ? I< 07:01:40 00:00:00 [ipv6_addrconf] root 2 127 0.0 0.0 0 0 ? I< 07:01:40 00:00:00 [kstrp] root 2 130 0.0 0.0 0 0 ? I< 07:01:40 00:00:00 [kworker/u257:0] root 2 143 0.0 0.0 0 0 ? I< 07:01:40 00:00:00 [charger_manager] root 2 180 0.0 0.0 0 0 ? I< 07:01:41 00:00:00 [hv_vmbus_con] root 2 181 0.0 0.0 0 0 ? I< 07:01:41 00:00:00 [hv_pri_chan] root 2 190 0.0 0.0 0 0 ? I< 07:01:41 00:00:00 [hv_sub_chan] root 2 191 0.0 0.0 0 0 ? I< 07:01:41 00:00:00 [kworker/3:1H-kblockd] root 2 192 0.0 0.0 0 0 ? I< 07:01:41 00:00:00 [cryptd] root 2 224 0.0 0.0 0 0 ? S 07:01:41 00:00:00 [scsi_eh_2] root 2 225 0.0 0.0 0 0 ? S 07:01:41 00:00:00 [scsi_eh_3] root 2 230 0.0 0.0 0 0 ? I< 07:01:41 00:00:00 [scsi_tmf_3] root 2 231 0.0 0.0 0 0 ? I< 07:01:41 00:00:00 [scsi_tmf_2] root 2 233 0.0 0.0 0 0 ? I< 07:01:41 00:00:00 [storvsc_error_w] root 2 238 0.0 0.0 0 0 ? I< 07:01:41 00:00:00 [storvsc_error_w] root 2 242 0.0 0.0 0 0 ? I< 07:01:41 00:00:00 [kworker/0:1H-kblockd] root 2 276 0.0 0.0 0 0 ? I< 07:01:43 00:00:00 [raid5wq] root 2 302 0.0 0.0 0 0 ? I< 07:01:43 00:00:00 [kworker/2:1H-kblockd] root 2 317 0.0 0.0 0 0 ? I< 07:01:43 00:00:00 [kworker/1:1H-kblockd] root 2 318 0.1 0.0 0 0 ? S 07:01:43 00:00:03 [jbd2/sda1-8] root 2 319 0.0 0.0 0 0 ? I< 07:01:43 00:00:00 [ext4-rsv-conver] root 1 421 0.0 0.0 19356 5660 ? Ss 07:01:45 00:00:01 /lib/systemd/systemd-udevd root 2 451 0.0 0.0 0 0 ? S 07:01:46 00:00:00 [hv_balloon] root 2 548 0.0 0.0 0 0 ? I< 07:01:47 00:00:00 [kaluad] root 2 549 0.0 0.0 0 0 ? I< 07:01:47 00:00:00 [kmpath_rdacd] root 2 550 0.0 0.0 0 0 ? I< 07:01:47 00:00:00 [kmpathd] root 2 551 0.0 0.0 0 0 ? I< 07:01:47 00:00:00 [kmpath_handlerd] root 1 552 0.0 0.1 280136 17940 ? SLsl 07:01:47 00:00:00 /sbin/multipathd -d -s root 2 563 0.0 0.0 0 0 ? S< 07:01:47 00:00:00 [loop0] root 2 566 0.0 0.0 0 0 ? S< 07:01:47 00:00:00 [loop1] root 2 568 0.0 0.0 0 0 ? S< 07:01:47 00:00:00 [loop2] root 1 779 0.0 0.0 241128 9528 ? Ssl 07:01:56 00:00:00 /usr/lib/accountsservice/accounts-daemon root 1 785 0.0 0.0 8536 2852 ? Ss 07:01:56 00:00:00 /usr/sbin/cron -f message+ 1 786 0.2 0.0 7732 4624 ? Ss 07:01:56 00:00:05 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only root 1 794 0.0 0.0 81828 3836 ? Ssl 07:01:56 00:00:00 /usr/sbin/irqbalance --foreground root 1 797 0.0 0.1 29264 17892 ? Ss 07:01:56 00:00:00 /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers root 1 808 0.1 0.2 1298740 34920 ? Ssl 07:01:56 00:00:03 /usr/lib/snapd/snapd root 1 809 0.1 0.0 16780 7548 ? Ss 07:01:56 00:00:02 /lib/systemd/systemd-logind daemon 1 811 0.0 0.0 3792 2144 ? Ss 07:01:56 00:00:00 /usr/sbin/atd -f root 1 851 0.0 0.0 7352 2160 ttyS0 Ss+ 07:01:56 00:00:00 /sbin/agetty -o -p -- \u --keep-baud 115200,38400,9600 ttyS0 vt220 root 1 883 0.0 0.0 5828 1756 tty1 Ss+ 07:01:57 00:00:00 /sbin/agetty -o -p -- \u --noclear tty1 linux root 1 885 0.0 0.1 108100 20816 ? Ssl 07:01:57 00:00:00 /usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal root 1 886 0.0 0.0 237748 9872 ? Ssl 07:01:57 00:00:00 /usr/lib/policykit-1/polkitd --no-debug root 1 1569 0.0 0.0 13800 6700 ? Ss 07:06:17 00:00:00 sshd: ubuntu [priv] ubuntu 1 1583 0.0 0.0 18568 9608 ? Ss 07:06:17 00:00:00 /lib/systemd/systemd --user ubuntu 1583 1593 0.0 0.0 169760 4552 ? S 07:06:17 00:00:00 (sd-pam) ubuntu 1569 1672 0.0 0.0 13932 5344 ? S 07:06:18 00:00:00 sshd: ubuntu@pts/0 root 2 14571 0.0 0.0 0 0 ? I 07:13:08 00:00:00 [kworker/3:0-rcu_par_gp] root 2 14580 0.0 0.0 0 0 ? I 07:13:08 00:00:00 [kworker/2:0-rcu_par_gp] root 2 14661 0.0 0.0 0 0 ? I 07:13:25 00:00:00 [kworker/u256:2-events_power_efficient] ubuntu 1672 14943 0.0 0.0 2608 596 pts/0 Ss+ 07:15:47 00:00:00 /bin/sh -c /usr/bin/python3 /home/ubuntu/.ansible/tmp/ansible-tmp-1647414947.55-107819321053808/AnsiballZ_command.py && sleep 0 ubuntu 14943 14944 0.0 0.1 28116 17820 pts/0 S+ 07:15:47 00:00:00 /usr/bin/python3 /home/ubuntu/.ansible/tmp/ansible-tmp-1647414947.55-107819321053808/AnsiballZ_command.py ubuntu 14944 14946 0.0 0.0 2608 592 pts/0 S+ 07:15:47 00:00:00 /bin/sh -c # workaround get-pip breaking when setuptools installed # ref: https://github.com/pypa/pip/issues/10742 sudo -H -E pip3 uninstall setuptools -y || true ./unstack.sh > /dev/null 2>&1 ./stack.sh > /dev/null 2>&1 ubuntu 1583 14950 0.0 0.0 7348 4380 ? Ss 07:15:48 00:00:00 /usr/bin/dbus-daemon --session --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only root 2 16199 0.0 0.0 2488 576 ? S 07:15:50 00:00:00 bpfilter_umh root 2 17329 0.0 0.0 0 0 ? I 07:15:52 00:00:00 [kworker/0:0-rcu_par_gp] root 2 17330 0.0 0.0 0 0 ? I 07:15:52 00:00:00 [kworker/u256:3-events_power_efficient] ubuntu 14946 17702 0.5 0.0 15724 10872 pts/0 S+ 07:15:54 00:00:08 bash ./stack.sh ubuntu 17702 19964 0.0 0.0 10640 3252 pts/0 S+ 07:15:57 00:00:00 bash ./stack.sh ubuntu 17702 19969 0.0 0.0 10640 3256 pts/0 S+ 07:15:57 00:00:00 bash ./stack.sh ubuntu 19964 19971 0.1 0.0 17516 11116 pts/0 S+ 07:15:57 00:00:02 /usr/bin/python3.8 /home/ubuntu/devstack/tools/outfilter.py -v --no-timestamp -o /opt/stack/logs/stack.sh.log.2022-03-16-071558 ubuntu 19969 19976 0.0 0.0 17384 11292 pts/0 S+ 07:15:57 00:00:00 /usr/bin/python3.8 /home/ubuntu/devstack/tools/outfilter.py -o /opt/stack/logs/stack.sh.log.2022-03-16-071558.summary.2022-03-16-071558 uuidd 1 25022 0.0 0.0 9744 1068 ? Ss 07:17:16 00:00:00 /usr/sbin/uuidd --socket-activation root 1 25090 0.0 0.0 8136 3804 ? Ss 07:17:17 00:00:00 /usr/sbin/conntrackd -C /etc/conntrackd/conntrackd.conf root 1 25328 0.0 0.0 15300 8732 ? Ss 07:17:21 00:00:00 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock haproxy 25328 25329 0.0 0.0 236496 2972 ? Sl 07:17:21 00:00:00 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock root 1 26193 0.0 0.0 12172 6408 ? Ss 07:17:32 00:00:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups pcp 1 28109 0.1 0.0 8356 4416 ? Ss 07:17:43 00:00:02 /usr/lib/pcp/bin/pmcd root 28109 28112 0.0 0.0 7736 3908 ? S 07:17:43 00:00:00 /var/lib/pcp/pmdas/root/pmdaroot root 28112 28113 0.0 0.0 8312 4996 ? S 07:17:43 00:00:00 /var/lib/pcp/pmdas/proc/pmdaproc -d 3 root 28112 28114 0.0 0.0 7212 1612 ? S 07:17:43 00:00:00 /var/lib/pcp/pmdas/xfs/pmdaxfs -d 11 root 28112 28115 0.3 0.0 8240 5040 ? S 07:17:43 00:00:04 /var/lib/pcp/pmdas/linux/pmdalinux root 28112 28116 0.0 0.0 7272 3792 ? S 07:17:43 00:00:00 /var/lib/pcp/pmdas/kvm/pmdakvm -d 95 systemd+ 1 39277 0.0 0.0 26740 7952 ? Ss 07:17:59 00:00:00 /lib/systemd/systemd-networkd systemd+ 1 39291 0.0 0.0 23896 10444 ? Ss 07:17:59 00:00:00 /lib/systemd/systemd-resolved systemd+ 1 39423 0.0 0.0 90224 6232 ? Ssl 07:18:00 00:00:00 /lib/systemd/systemd-timesyncd syslog 1 44136 0.2 0.0 224344 4176 ? Ssl 07:18:14 00:00:02 /usr/sbin/rsyslogd -n -iNONE root 1 53572 0.5 0.5 109180 66608 ? S mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: mtu 1500 qdisc mq state UP group default qlen 1000 link/ether fa:16:3e:72:24:7e brd ff:ff:ff:ff:ff:ff inet 10.222.0.39/22 brd 10.222.3.255 scope global dynamic eth0 valid_lft 85158sec preferred_lft 85158sec inet6 fe80::f816:3eff:fe72:247e/64 scope link valid_lft forever preferred_lft forever 3: eth1: mtu 1500 qdisc mq master ovs-system state UP group default qlen 1000 link/ether fa:16:3e:95:df:b0 brd ff:ff:ff:ff:ff:ff inet6 fe80::f816:3eff:fe95:dfb0/64 scope link valid_lft forever preferred_lft forever 4: ovs-system: mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 02:29:10:8e:75:98 brd ff:ff:ff:ff:ff:ff 5: br-int: mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 0a:f5:f9:5c:07:44 brd ff:ff:ff:ff:ff:ff 6: br-data: mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether ae:f2:0f:19:97:4e brd ff:ff:ff:ff:ff:ff inet 192.168.0.36/22 brd 192.168.3.255 scope global br-data valid_lft forever preferred_lft forever inet 172.26.0.1/24 scope global br-data valid_lft forever preferred_lft forever inet6 fe80::acf2:fff:fe19:974e/64 scope link valid_lft forever preferred_lft forever ip route -------- default via 10.222.0.1 dev eth0 proto dhcp src 10.222.0.39 metric 100 10.222.0.0/22 dev eth0 proto kernel scope link src 10.222.0.39 169.254.169.254 via 10.222.0.12 dev eth0 proto dhcp src 10.222.0.39 metric 100 172.26.0.0/24 dev br-data proto kernel scope link src 172.26.0.1 192.168.0.0/22 dev br-data proto kernel scope link src 192.168.0.36 192.168.0.0/22 dev eth0 proto dhcp scope link src 10.222.0.39 metric 100 ip -6 route ----------- ::1 dev lo proto kernel metric 256 pref medium fe80::/64 dev eth1 proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium fe80::/64 dev br-data proto kernel metric 256 pref medium ip netns -------- qdhcp-50678852-434d-42ea-89c8-6de80f17a182 (id: 2) qrouter-6b619feb-1293-42be-83d3-54c3bcfb091c (id: 1) qdhcp-4c1d4356-765c-4315-b049-75a92d4ae0f8 (id: 0) sudo ip netns exec qdhcp-50678852-434d-42ea-89c8-6de80f17a182 ip neigh ---------------------------------------------------------------------- sudo ip netns exec qdhcp-50678852-434d-42ea-89c8-6de80f17a182 ip addr --------------------------------------------------------------------- 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 10: tap505e47bf-3d: mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether fa:16:3e:84:50:86 brd ff:ff:ff:ff:ff:ff inet 192.168.233.2/24 brd 192.168.233.255 scope global tap505e47bf-3d valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe84:5086/64 scope link valid_lft forever preferred_lft forever sudo ip netns exec qdhcp-50678852-434d-42ea-89c8-6de80f17a182 ip route ---------------------------------------------------------------------- default via 192.168.233.1 dev tap505e47bf-3d proto static 192.168.233.0/24 dev tap505e47bf-3d proto kernel scope link src 192.168.233.2 sudo ip netns exec qdhcp-50678852-434d-42ea-89c8-6de80f17a182 ip -6 route ------------------------------------------------------------------------- fe80::/64 dev tap505e47bf-3d proto kernel metric 256 pref medium sudo ip netns exec qrouter-6b619feb-1293-42be-83d3-54c3bcfb091c ip neigh ------------------------------------------------------------------------ fe80::acf2:fff:fe19:974e dev qg-8e26795f-65 lladdr ae:f2:0f:19:97:4e STALE fe80::f816:3eff:fe0e:4d1b dev qg-8e26795f-65 lladdr fa:16:3e:0e:4d:1b STALE fe80::f816:3eff:fe72:247e dev qg-8e26795f-65 lladdr fa:16:3e:72:24:7e STALE fe80::328d:99ff:fe30:c30 dev qg-8e26795f-65 lladdr 30:8d:99:30:0c:30 STALE sudo ip netns exec qrouter-6b619feb-1293-42be-83d3-54c3bcfb091c ip addr ----------------------------------------------------------------------- 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 8: qr-bef8762e-4c: mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether fa:16:3e:ae:20:70 brd ff:ff:ff:ff:ff:ff inet 172.20.0.1/26 brd 172.20.0.63 scope global qr-bef8762e-4c valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:feae:2070/64 scope link valid_lft forever preferred_lft forever 9: qg-8e26795f-65: mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether fa:16:3e:7c:1a:38 brd ff:ff:ff:ff:ff:ff inet 172.26.0.175/24 brd 172.26.0.255 scope global qg-8e26795f-65 valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe7c:1a38/64 scope link valid_lft forever preferred_lft forever sudo ip netns exec qrouter-6b619feb-1293-42be-83d3-54c3bcfb091c ip route ------------------------------------------------------------------------ default via 172.26.0.1 dev qg-8e26795f-65 proto static 172.20.0.0/26 dev qr-bef8762e-4c proto kernel scope link src 172.20.0.1 172.26.0.0/24 dev qg-8e26795f-65 proto kernel scope link src 172.26.0.175 sudo ip netns exec qrouter-6b619feb-1293-42be-83d3-54c3bcfb091c ip -6 route --------------------------------------------------------------------------- fe80::/64 dev qr-bef8762e-4c proto kernel metric 256 pref medium fe80::/64 dev qg-8e26795f-65 proto kernel metric 256 pref medium sudo ip netns exec qdhcp-4c1d4356-765c-4315-b049-75a92d4ae0f8 ip neigh ---------------------------------------------------------------------- sudo ip netns exec qdhcp-4c1d4356-765c-4315-b049-75a92d4ae0f8 ip addr --------------------------------------------------------------------- 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 7: tap0b92d56b-0d: mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether fa:16:3e:f7:cc:1a brd ff:ff:ff:ff:ff:ff inet 172.20.0.2/26 brd 172.20.0.63 scope global tap0b92d56b-0d valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fef7:cc1a/64 scope link valid_lft forever preferred_lft forever sudo ip netns exec qdhcp-4c1d4356-765c-4315-b049-75a92d4ae0f8 ip route ---------------------------------------------------------------------- default via 172.20.0.1 dev tap0b92d56b-0d proto static 172.20.0.0/26 dev tap0b92d56b-0d proto kernel scope link src 172.20.0.2 sudo ip netns exec qdhcp-4c1d4356-765c-4315-b049-75a92d4ae0f8 ip -6 route ------------------------------------------------------------------------- fe80::/64 dev tap0b92d56b-0d proto kernel metric 256 pref medium Open vSwitch Dump ================= sudo ovs-vsctl show ------------------- 02bf8f4e-d1b0-4b5c-8c10-90e45b2685e5 Manager "ptcp:6640:127.0.0.1" is_connected: true Bridge br-data Controller "tcp:127.0.0.1:6633" is_connected: true fail_mode: secure datapath_type: system Port phy-br-data Interface phy-br-data type: patch options: {peer=int-br-data} Port br-data Interface br-data type: internal Port eth1 Interface eth1 Bridge br-int Controller "tcp:127.0.0.1:6633" is_connected: true fail_mode: secure datapath_type: system Port tap505e47bf-3d tag: 3 Interface tap505e47bf-3d type: internal Port tap0b92d56b-0d tag: 1 Interface tap0b92d56b-0d type: internal Port qr-bef8762e-4c tag: 1 Interface qr-bef8762e-4c type: internal Port int-br-data Interface int-br-data type: patch options: {peer=phy-br-data} Port br-int Interface br-int type: internal Port qg-8e26795f-65 tag: 2 Interface qg-8e26795f-65 type: internal ovs_version: "2.13.5" sudo ovs-ofctl --protocols=OpenFlow10,OpenFlow11,OpenFlow12,OpenFlow13,OpenFlow14,OpenFlow15 show br-data --------------------------------------------------------------------------------------------------------- OFPT_FEATURES_REPLY (OF1.4) (xid=0x2): dpid:0000aef20f19974e n_tables:254, n_buffers:0 capabilities: FLOW_STATS TABLE_STATS PORT_STATS GROUP_STATS QUEUE_STATS BUNDLES OFPST_PORT_DESC reply (OF1.4) (xid=0x3): 1(eth1): addr:fa:16:3e:95:df:b0 config: 0 state: LIVE speed: 0 Mbps now, 0 Mbps max 2(phy-br-data): addr:9e:b9:94:76:af:60 config: 0 state: LIVE speed: 0 Mbps now, 0 Mbps max LOCAL(br-data): addr:ae:f2:0f:19:97:4e config: 0 state: LIVE speed: 0 Mbps now, 0 Mbps max OFPT_GET_CONFIG_REPLY (OF1.4) (xid=0x5): frags=normal miss_send_len=0 sudo ovs-ofctl --protocols=OpenFlow10,OpenFlow11,OpenFlow12,OpenFlow13,OpenFlow14,OpenFlow15 show br-int -------------------------------------------------------------------------------------------------------- OFPT_FEATURES_REPLY (OF1.4) (xid=0x2): dpid:00000af5f95c0744 n_tables:254, n_buffers:0 capabilities: FLOW_STATS TABLE_STATS PORT_STATS GROUP_STATS QUEUE_STATS BUNDLES OFPST_PORT_DESC reply (OF1.4) (xid=0x3): 1(int-br-data): addr:c6:22:36:e3:13:9f config: 0 state: LIVE speed: 0 Mbps now, 0 Mbps max 2(tap0b92d56b-0d): addr:fa:16:3e:f7:cc:1a config: 0 state: LIVE speed: 0 Mbps now, 0 Mbps max 3(qr-bef8762e-4c): addr:fa:16:3e:ae:20:70 config: 0 state: LIVE speed: 0 Mbps now, 0 Mbps max 4(qg-8e26795f-65): addr:fa:16:3e:7c:1a:38 config: 0 state: LIVE speed: 0 Mbps now, 0 Mbps max 5(tap505e47bf-3d): addr:fa:16:3e:84:50:86 config: 0 state: LIVE speed: 0 Mbps now, 0 Mbps max LOCAL(br-int): addr:0a:f5:f9:5c:07:44 config: PORT_DOWN state: LINK_DOWN speed: 0 Mbps now, 0 Mbps max OFPT_GET_CONFIG_REPLY (OF1.4) (xid=0x5): frags=normal miss_send_len=0 sudo ovs-ofctl --protocols=OpenFlow10,OpenFlow11,OpenFlow12,OpenFlow13,OpenFlow14,OpenFlow15 dump-ports-desc br-data -------------------------------------------------------------------------------------------------------------------- OFPST_PORT_DESC reply (OF1.4) (xid=0x2): 1(eth1): addr:fa:16:3e:95:df:b0 config: 0 state: LIVE speed: 0 Mbps now, 0 Mbps max 2(phy-br-data): addr:9e:b9:94:76:af:60 config: 0 state: LIVE speed: 0 Mbps now, 0 Mbps max LOCAL(br-data): addr:ae:f2:0f:19:97:4e config: 0 state: LIVE speed: 0 Mbps now, 0 Mbps max sudo ovs-ofctl --protocols=OpenFlow10,OpenFlow11,OpenFlow12,OpenFlow13,OpenFlow14,OpenFlow15 dump-ports-desc br-int ------------------------------------------------------------------------------------------------------------------- OFPST_PORT_DESC reply (OF1.4) (xid=0x2): 1(int-br-data): addr:c6:22:36:e3:13:9f config: 0 state: LIVE speed: 0 Mbps now, 0 Mbps max 2(tap0b92d56b-0d): addr:fa:16:3e:f7:cc:1a config: 0 state: LIVE speed: 0 Mbps now, 0 Mbps max 3(qr-bef8762e-4c): addr:fa:16:3e:ae:20:70 config: 0 state: LIVE speed: 0 Mbps now, 0 Mbps max 4(qg-8e26795f-65): addr:fa:16:3e:7c:1a:38 config: 0 state: LIVE speed: 0 Mbps now, 0 Mbps max 5(tap505e47bf-3d): addr:fa:16:3e:84:50:86 config: 0 state: LIVE speed: 0 Mbps now, 0 Mbps max LOCAL(br-int): addr:0a:f5:f9:5c:07:44 config: PORT_DOWN state: LINK_DOWN speed: 0 Mbps now, 0 Mbps max sudo ovs-ofctl --protocols=OpenFlow10,OpenFlow11,OpenFlow12,OpenFlow13,OpenFlow14,OpenFlow15 dump-ports br-data --------------------------------------------------------------------------------------------------------------- OFPST_PORT reply (OF1.4) (xid=0x2): 3 ports port LOCAL: rx pkts=220, bytes=52257, drop=0, errs=0, frame=0, over=0, crc=0 tx pkts=12, bytes=936, drop=0, errs=0, coll=0 duration=412.134s port 1: rx pkts=1901, bytes=367125, drop=0, errs=0, frame=0, over=0, crc=0 tx pkts=69, bytes=6234, drop=0, errs=0, coll=0 duration=407.917s port 2: rx pkts=28, bytes=2056, drop=?, errs=?, frame=?, over=?, crc=? tx pkts=148, bytes=39309, drop=?, errs=?, coll=? duration=296.229s sudo ovs-ofctl --protocols=OpenFlow10,OpenFlow11,OpenFlow12,OpenFlow13,OpenFlow14,OpenFlow15 dump-ports br-int -------------------------------------------------------------------------------------------------------------- OFPST_PORT reply (OF1.4) (xid=0x2): 6 ports port 4: rx pkts=105, bytes=28920, drop=0, errs=0, frame=0, over=0, crc=0 tx pkts=19, bytes=1418, drop=0, errs=0, coll=0 duration=248.592s port LOCAL: rx pkts=0, bytes=0, drop=133, errs=0, frame=0, over=0, crc=0 tx pkts=0, bytes=0, drop=0, errs=0, coll=0 duration=412.351s port 1: rx pkts=148, bytes=39309, drop=?, errs=?, frame=?, over=?, crc=? tx pkts=28, bytes=2056, drop=?, errs=?, coll=? duration=296.292s port 5: rx pkts=0, bytes=0, drop=0, errs=0, frame=0, over=0, crc=0 tx pkts=5, bytes=446, drop=0, errs=0, coll=0 duration=105.869s port 2: rx pkts=6, bytes=304, drop=0, errs=0, frame=0, over=0, crc=0 tx pkts=5, bytes=446, drop=0, errs=0, coll=0 duration=269.888s port 3: rx pkts=0, bytes=0, drop=0, errs=0, frame=0, over=0, crc=0 tx pkts=13, bytes=998, drop=0, errs=0, coll=0 duration=252.478s sudo ovs-ofctl --protocols=OpenFlow10,OpenFlow11,OpenFlow12,OpenFlow13,OpenFlow14,OpenFlow15 dump-flows br-data --------------------------------------------------------------------------------------------------------------- OFPST_FLOW reply (OF1.4) (xid=0x2): cookie=0xeeeef391b4575b90, duration=268.552s, table=0, n_packets=8, n_bytes=568, priority=4,in_port=2,dl_vlan=1 actions=set_field:4234->vlan_vid,NORMAL cookie=0xeeeef391b4575b90, duration=248.525s, table=0, n_packets=18, n_bytes=1308, priority=4,in_port=2,dl_vlan=2 actions=pop_vlan,NORMAL cookie=0xeeeef391b4575b90, duration=104.447s, table=0, n_packets=2, n_bytes=180, priority=4,in_port=2,dl_vlan=3 actions=set_field:4224->vlan_vid,NORMAL cookie=0xeeeef391b4575b90, duration=296.228s, table=0, n_packets=0, n_bytes=0, priority=2,in_port=2 actions=drop cookie=0xeeeef391b4575b90, duration=296.312s, table=0, n_packets=148, n_bytes=39309, priority=0 actions=NORMAL sudo ovs-ofctl --protocols=OpenFlow10,OpenFlow11,OpenFlow12,OpenFlow13,OpenFlow14,OpenFlow15 dump-flows br-int -------------------------------------------------------------------------------------------------------------- OFPST_FLOW reply (OF1.4) (xid=0x2): cookie=0xc47d879ba0963228, duration=297.531s, table=0, n_packets=0, n_bytes=0, priority=65535,dl_vlan=4095 actions=drop cookie=0xc47d879ba0963228, duration=268.560s, table=0, n_packets=0, n_bytes=0, priority=3,in_port=1,dl_vlan=138 actions=set_field:4097->vlan_vid,goto_table:59 cookie=0xc47d879ba0963228, duration=248.536s, table=0, n_packets=105, n_bytes=30390, priority=3,in_port=1,vlan_tci=0x0000/0x1fff actions=push_vlan:0x8100,set_field:4098->vlan_vid,goto_table:59 cookie=0xc47d879ba0963228, duration=104.459s, table=0, n_packets=0, n_bytes=0, priority=3,in_port=1,dl_vlan=128 actions=set_field:4099->vlan_vid,goto_table:59 cookie=0xc47d879ba0963228, duration=296.246s, table=0, n_packets=43, n_bytes=8919, priority=2,in_port=1 actions=drop cookie=0xc47d879ba0963228, duration=297.541s, table=0, n_packets=42, n_bytes=3308, priority=0 actions=goto_table:59 cookie=0xc47d879ba0963228, duration=297.544s, table=23, n_packets=0, n_bytes=0, priority=0 actions=drop cookie=0xc47d879ba0963228, duration=297.533s, table=24, n_packets=0, n_bytes=0, priority=0 actions=drop cookie=0xc47d879ba0963228, duration=297.525s, table=30, n_packets=0, n_bytes=0, priority=0 actions=goto_table:59 cookie=0xc47d879ba0963228, duration=297.523s, table=31, n_packets=0, n_bytes=0, priority=0 actions=goto_table:59 cookie=0xc47d879ba0963228, duration=297.539s, table=59, n_packets=147, n_bytes=33698, priority=0 actions=goto_table:60 cookie=0xc47d879ba0963228, duration=266.525s, table=60, n_packets=2, n_bytes=180, priority=100,in_port=2 actions=set_field:0x2->reg5,set_field:0x1->reg6,resubmit(,73) cookie=0xc47d879ba0963228, duration=250.484s, table=60, n_packets=6, n_bytes=388, priority=100,in_port=3 actions=set_field:0x3->reg5,set_field:0x1->reg6,resubmit(,73) cookie=0xc47d879ba0963228, duration=246.482s, table=60, n_packets=18, n_bytes=1308, priority=100,in_port=4 actions=set_field:0x4->reg5,set_field:0x2->reg6,resubmit(,73) cookie=0xc47d879ba0963228, duration=104.381s, table=60, n_packets=2, n_bytes=180, priority=100,in_port=5 actions=set_field:0x5->reg5,set_field:0x3->reg6,resubmit(,73) cookie=0xc47d879ba0963228, duration=297.536s, table=60, n_packets=119, n_bytes=31642, priority=3 actions=NORMAL cookie=0xc47d879ba0963228, duration=297.529s, table=62, n_packets=0, n_bytes=0, priority=3 actions=NORMAL cookie=0xc47d879ba0963228, duration=291.255s, table=71, n_packets=0, n_bytes=0, reset_counts priority=110,ct_state=+trk actions=ct_clear,resubmit(,71) cookie=0xc47d879ba0963228, duration=291.305s, table=71, n_packets=0, n_bytes=0, reset_counts priority=0 actions=drop cookie=0xc47d879ba0963228, duration=291.295s, table=72, n_packets=0, n_bytes=0, reset_counts priority=0 actions=drop cookie=0xc47d879ba0963228, duration=266.525s, table=73, n_packets=2, n_bytes=180, priority=80,reg5=0x2 actions=resubmit(,94) cookie=0xc47d879ba0963228, duration=250.484s, table=73, n_packets=6, n_bytes=388, priority=80,reg5=0x3 actions=resubmit(,94) cookie=0xc47d879ba0963228, duration=246.482s, table=73, n_packets=18, n_bytes=1308, priority=80,reg5=0x4 actions=resubmit(,94) cookie=0xc47d879ba0963228, duration=104.381s, table=73, n_packets=2, n_bytes=180, priority=80,reg5=0x5 actions=resubmit(,94) cookie=0xc47d879ba0963228, duration=291.285s, table=73, n_packets=0, n_bytes=0, reset_counts priority=0 actions=drop cookie=0xc47d879ba0963228, duration=291.275s, table=81, n_packets=0, n_bytes=0, reset_counts priority=0 actions=drop cookie=0xc47d879ba0963228, duration=291.265s, table=82, n_packets=0, n_bytes=0, reset_counts priority=0 actions=drop cookie=0xc47d879ba0963228, duration=291.235s, table=91, n_packets=0, n_bytes=0, reset_counts priority=1 actions=resubmit(,94) cookie=0xc47d879ba0963228, duration=291.225s, table=92, n_packets=0, n_bytes=0, reset_counts priority=0 actions=drop cookie=0xc47d879ba0963228, duration=291.215s, table=93, n_packets=0, n_bytes=0, reset_counts priority=0 actions=drop cookie=0xc47d879ba0963228, duration=291.244s, table=94, n_packets=28, n_bytes=2056, reset_counts priority=1 actions=NORMAL IP Tables Dump ============== sudo iptables --line-numbers -L -nv -t filter --------------------------------------------- Chain INPUT (policy ACCEPT 739K packets, 3720M bytes) num pkts bytes target prot opt in out source destination Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) num pkts bytes target prot opt in out source destination Chain OUTPUT (policy ACCEPT 675K packets, 2704M bytes) num pkts bytes target prot opt in out source destination sudo iptables --line-numbers -L -nv -t nat ------------------------------------------ Chain PREROUTING (policy ACCEPT 100 packets, 32280 bytes) num pkts bytes target prot opt in out source destination Chain INPUT (policy ACCEPT 98 packets, 32144 bytes) num pkts bytes target prot opt in out source destination Chain OUTPUT (policy ACCEPT 56 packets, 3512 bytes) num pkts bytes target prot opt in out source destination Chain POSTROUTING (policy ACCEPT 56 packets, 3512 bytes) num pkts bytes target prot opt in out source destination 1 0 0 MASQUERADE all -- * eth0 172.26.0.0/24 0.0.0.0/0 sudo iptables --line-numbers -L -nv -t mangle --------------------------------------------- Chain PREROUTING (policy ACCEPT 258K packets, 3257M bytes) num pkts bytes target prot opt in out source destination Chain INPUT (policy ACCEPT 258K packets, 3257M bytes) num pkts bytes target prot opt in out source destination Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) num pkts bytes target prot opt in out source destination Chain OUTPUT (policy ACCEPT 258K packets, 2611M bytes) num pkts bytes target prot opt in out source destination Chain POSTROUTING (policy ACCEPT 258K packets, 2611M bytes) num pkts bytes target prot opt in out source destination EB Tables Dump ============== sudo ebtables -t filter -L -------------------------- Bridge table: filter Bridge chain: INPUT, entries: 0, policy: ACCEPT Bridge chain: FORWARD, entries: 0, policy: ACCEPT Bridge chain: OUTPUT, entries: 0, policy: ACCEPT sudo ebtables -t nat -L ----------------------- Bridge table: nat Bridge chain: PREROUTING, entries: 0, policy: ACCEPT Bridge chain: OUTPUT, entries: 0, policy: ACCEPT Bridge chain: POSTROUTING, entries: 0, policy: ACCEPT Compute consoles ================ nova-compute Guru Meditation Report =================================== Skipping as nova-compute does not appear to be running neutron-dhcp-agent Guru Meditation Report ========================================= 111736 killall -e -USR2 neutron-dhcp-agent ----------------------------------- guru meditation report in neutron-dhcp-agent log neutron-l3-agent Guru Meditation Report ======================================= 112111 killall -e -USR2 neutron-l3-agent --------------------------------- guru meditation report in neutron-l3-agent log neutron-linuxbridge-agent Guru Meditation Report ================================================ Skipping as neutron-linuxbridge-agent does not appear to be running neutron-metadata-agent Guru Meditation Report ============================================= 112468 112956 112957 killall -e -USR2 neutron-metadata-agent --------------------------------------- guru meditation report in neutron-metadata-agent log neutron-openvswitch-agent Guru Meditation Report ================================================ 111373 killall -e -USR2 neutron-openvswitch-agent ------------------------------------------ guru meditation report in neutron-openvswitch-agent log cinder-volume Guru Meditation Report ==================================== 118412 118517 killall -e -USR2 cinder-volume ------------------------------ guru meditation report in cinder-volume log