Linux uses the weak host model, with apparently no way to control this.
This, combined with a somewhat complicated network setup, lead to a weird situation where two boxes can ping each other, have full access to other networks, but cannot create a TCP connection between each other.
Server A is multihomed with two physical network interfaces. Let's say eth0 is on 192.168.1.10/24 and eth1 is on 192.168.2.10/24.
Server B has a single network interface 192.168.2.20/24.
There is a router that permits all traffic between the two subnets, and permits a limited amount of traffic between each subnet and the main engineering LAN.
Right, now server B needs to make a connection to server A on 192.168.1.10. Server B doesn't have a direct route to server A, so it sends the TCP SYN packet to the router for 192.168.2.0/24.
Server A receives this packet on eth0. Because the weak host model permits packets to be sent from any interface, it uses eth1 (configured with IP address 192.168.2.10) to send the SYN/ACK with a source IP address of 192.168.1.10. This SYN/ACK packet bypasses the router.
Server B receives the SYN/ACK. It now has to send an ACK to server A. Again, since it doesn't have a direct route it has to send the ACK via the router.
The router eats the ACK. Presumably the router is running a stateful firewall, and so because it hasn't seen the SYN/ACK it "knows" that server B shouldn't have sent the ACK yet.
Because server A hasn't received the ACK it doesn't know that the connection has been established, and so after a few retransmissions on both ends it kills off the connection.
I eventually had to use the Linux ip command to set up a static route on server A specifically for server B. Oh, and just to make tracking down this problem more fun, the exact same configuration was working perfectly the previous day.
