In a payment card association such as Visa, each time a consumer pays by card, the bank of the merchant (acquirer) pays an interchange fee (IF) to the bank of the cardholder (issuer) to carry out the transaction. This paper studies the determinants of socially and privately optimal IFs in a card scheme where services are provided by a monopoly issuer and perfectly competitive acquirers to heterogeneous consumers and merchants. Different from the literature, we distinguish card membership from card usage decisions (and fees). In doing so, we reveal the implications of an asymmetry between consumers and merchants: the card usage decision at a point of sale is delegated to cardholders since merchants are not allowed to turn down cards once they are affiliated with a card network. We show that this asymmetry is sufficient to induce the card association to set a higher IF than the socially optimal IF, and thus to distort the structure of user fees by leading to too low card usage fees at the expense of too high merchant fees. Hence, cap regulations on IFs can improve the welfare. These qualitative results are robust to imperfect issuer competition, imperfect acquirer competition, and to other factors affecting final demands, such as elastic consumer participation or strategic card acceptance to attract consumers.