We're updating the issue view to help you get more done. 

fip64 failure seen on ost gate

Description

http://logs.openstack.org/87/199387/102/check/networking-midonet-tempest-aio-ml2/fcc3ddb/logs/

the job runs the test_fip64 test case twice. (temporarily for neutron-tempest-plugin transition)
the second run is consistently failing.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 2018.01.16 11:54:37.787 DEBUG [devices-service] vpp-controller - FIP64 entry added: Fip64 [fixedIp=10.1.0.2 floatingIp=2001:db8:0:0:0:0:0:e natPool=100.64.0.0/10 portId=ba8ce6b1-244a-4f01-961a-b76b317a817d routerId=520fa3fb-3872-4a7a-89e5-ce340f556f28] 2018.01.16 11:54:37.787 DEBUG [devices-service] vpp-controller - Subscribing to downlink port ba8ce6b1-244a-4f01-961a-b76b317a817d with VRF 3 2018.01.16 11:54:37.788 DEBUG [devices-service] vpp-controller - Port updated: RouterPort [id=ba8ce6b1-244a-4f01-961a-b76b317a817d active=false adminStateUp=true inboundFilters=[] outboundFilters=[] tunnelKey=2097218 portGroups=[] peerId=6928b635-7465-4c67-89cd-6507f62ce80f hostId=null interfaceName=nu ll vlanId=0 routerId=520fa3fb-3872-4a7a-89e5-ce340f556f28 portAddresses=[172.24.5.10/24, 2001:db8:0:0:0:0:0:c/64, 169.254.0.1/30] portAddress4=172.24.5.10/24 portAddress6=2001:db8:0:0:0:0:0:c/64 portMac=fa:16:3e:84:8a:16 routeIds=Set(08044f40-ff1f-4523-a5ba-fe3d1b7d7add, a931e326-6382-4096-96e4-d3155f0 a2d23, 08044f5d-caf8-440f-a5ba-fc7975501728, d1293b39-b728-49d9-86b7-049d549c235c, a931e326-4a96-4d02-96e4-d3155f0a245d)] 2018.01.16 11:54:37.788 DEBUG [devices-service] vpp-controller - Downlink port ba8ce6b1-244a-4f01-961a-b76b317a817d notification: CreateTunnel [port=ba8ce6b1-244a-4f01-961a-b76b317a817d vrf=3 vni=2097218 routerPortMac=fa:16:3e:84:8a:16] 2018.01.16 11:54:37.788 DEBUG [devices-service] vpp-controller - Downlink port ba8ce6b1-244a-4f01-961a-b76b317a817d notification: AssociateFip [port=ba8ce6b1-244a-4f01-961a-b76b317a817d vrf=3 floatingIp=2001:db8:0:0:0:0:0:e fixedIp=10.1.0.2 localIp=172.24.5.10/24 natPool=100.64.0.0/10] 2018.01.16 11:54:37.788 DEBUG [vpp-controller] vpp-controller - Creating downlink tunnel for port:ba8ce6b1-244a-4f01-961a-b76b317a817d VRF:3 VNI:2097218 2018.01.16 11:54:37.788 DEBUG [vpp-controller] vpp-controller - Started to execute VXLAN tunnel setup 2018.01.16 11:54:37.788 DEBUG [vpp-controller] vpp-controller - Sequence VXLAN tunnel setup: execute Create VXLAN tunnel 2018.01.16 11:54:37.795 DEBUG [vpp-controller] vpp-controller - Sequence VXLAN tunnel setup: completed execute Create VXLAN tunnel 2018.01.16 11:54:37.795 DEBUG [vpp-controller] vpp-controller - Sequence VXLAN tunnel setup: execute Set bridge domain for VXLAN device 2018.01.16 11:54:37.805 DEBUG [vpp-controller] vpp-controller - Sequence VXLAN tunnel setup: completed execute Set bridge domain for VXLAN device 2018.01.16 11:54:37.805 DEBUG [vpp-controller] vpp-controller - Sequence VXLAN tunnel setup: execute Create loopback interface 2018.01.16 11:54:37.825 DEBUG [vpp-controller] vpp-controller - Sequence VXLAN tunnel setup: completed execute Create loopback interface 2018.01.16 11:54:37.825 DEBUG [vpp-controller] vpp-controller - Sequence VXLAN tunnel setup: execute Set bridge domain for loopback device 2018.01.16 11:54:37.835 DEBUG [vpp-controller] vpp-controller - Sequence VXLAN tunnel setup: completed execute Set bridge domain for loopback device 2018.01.16 11:54:37.835 DEBUG [vpp-controller] vpp-controller - Sequence VXLAN tunnel setup: execute Move loopback interface to VRF 3 2018.01.16 11:54:37.845 DEBUG [vpp-controller] vpp-controller - Sequence VXLAN tunnel setup: completed execute Move loopback interface to VRF 3 2018.01.16 11:54:37.845 DEBUG [vpp-controller] vpp-controller - Sequence VXLAN tunnel setup: execute Add ARP neighbour for loop interface 2018.01.16 11:54:37.858 DEBUG [vpp-controller] vpp-controller - Sequence VXLAN tunnel setup: completed execute Add ARP neighbour for loop interface 2018.01.16 11:54:37.858 DEBUG [vpp-controller] vpp-controller - Completed execute of VXLAN tunnel setup 2018.01.16 11:54:37.858 DEBUG [vpp-controller] vpp-controller - Associating FIP at port ba8ce6b1-244a-4f01-961a-b76b317a817d (VRF 3, VNI 2097218): 2001:db8:0:0:0:0:0:e -> 10.1.0.2 2018.01.16 11:54:37.858 DEBUG [vpp-controller] vpp-controller - Allocated NAT pool at port ba8ce6b1-244a-4f01-961a-b76b317a817d is NatPool(100.64.0.1,100.127.255.254) 2018.01.16 11:54:37.858 DEBUG [vpp-controller] vpp-controller - Started to execute Fip64 setup 2018.01.16 11:54:37.858 DEBUG [vpp-controller] vpp-controller - Sequence Fip64 setup: execute fip64 2001:db8:0:0:0:0:0:e <-> 10.1.0.2 2018.01.16 11:54:37.867 WARN [vpp-controller] vpp-controller - Sequence Fip64 setup: execute failed fip64 2001:db8:0:0:0:0:0:e <-> 10.1.0.2: org.openvpp.jvpp.VppCallbackException: vppApi.fip64_add_reply failed with error code: -1 2018.01.16 11:54:37.867 WARN [vpp-controller] vpp-controller - Failed to execute Fip64 setup: org.openvpp.jvpp.VppCallbackException: vppApi.fip64_add_reply failed with error code: -1 2018.01.16 11:54:37.867 ERROR [vpp-controller] vpp-controller - Error on conveyor belt org.openvpp.jvpp.VppCallbackException: vppApi.fip64_add_reply failed with error code: -1

Status

Assignee

Unassigned

Reporter

Takashi Yamamoto

Priority

Major