Take the following network layout:
This shows two servers: server A has only one network interface, while server B has two. Now let's assume that server B is running a useful service on 192.168.2.11 which server A wants to access. Since server A is only connected to the 192.168.1.x network, to get at this service it has to use the router.
One would expect server B to work out that because the service is running on a different subnet to server A, that it would also have to use the router for any replies. With the strong host model (where an IP address belongs to a specific interface) this would work perfectly and everyone would be happy.
The problem here is that Linux uses the weak host model, where a server has a set of IP addresses and it can use any address on any interface. This causes all sorts of fun problems even when you're expecting it. In this case, it results in server B sending the replies via the 192.168.1.11 interface but with a source address of 192.168.2.11. Now, you may be thinking that this isn't really a problem because the packets are still getting to the right place, but you would be wrong. The router only sees half the traffic, and if it's running some sort of stateful firewall then it will get very unhappy and break your network connection.
I'd run into that exact problem some time ago with a test network at work and managed to work around it by adding several explicit routing instructions to the network config (which I'll dig out and post at some point). However, that's not all. Today a colleague found another way in which Linux breaks things.
Linux has an option called "rp_filter". Turning this on causes Linux to check the source address of each packet it receives against the list of addresses it knows it can route to via that interface. And, of course, it breaks the network setup I've shown above.
The problem this time is that the routing table on server B says that to get to 192.168.1.x it must use the top network, to get to 192.168.2.x it must use the bottom network, and for anything else it uses the top network. So when it gets a packet from server A to 192.168.2.11 it checks the source address of 192.168.1.10 against the list of routes for the bottom interface, fails to find a matching address, and discards the packet. Notably, it discards the packet without sending any kind of error back to the sender, making this a right pain to debug.
For added fun, it turns out that the distro we use changed the default for this parameter sometime in the past year or so. So all our test systems that were built ages ago work fine, while the shiny new system someone just built on site ends up with weird networking problems because of this!