For a comfortable fit while carrying an IWB holster, it is recommended that you wear pants two inches larger than normal to accommodate the holster. However, the pants often appear baggy and are especially large and uncomfortable when not using an IWB holster.
Deluth Training Company may have the answer. In their FireHose series of men's pants they offer "Self-Tailoring Travel Pants" with an expandable waistband. Rather than just an elastic sewn into the waistband, these are two overlapping bands of fabric that form the waist band, held together by internal elastic. The result is that the pants expand and contract invisibly.
At $54 they are a little pricey but they are very well made. The adjustability and added comfort is worth the price. Unfortunately, the do not offer a pleated version so pocket carry is limited to small pocket rockets.