If it is an ad-swap with even traffic levels then I don't think people will have a problem - often these can be sent without either of you using an affiliate link, and if this is the case then it doesn't matter where their traffic is going to.

The only problem you would have is a proper cross promotion where your partner wants to make commission as well as receiving traffic, so longer term it might be a good idea to develop these types of relationship and obviously develop a product too