Various secondary plant metabolites or phytochemicals, including polyphenols and carotenoids, have been associated with a variety of health benefits, such as reduced incidence of type 2 diabetes, cardiovascular diseases, and several types of cancer, most likely due to their involvement in ameliorating inflammation and oxidative stress. However, discrepancies exist between their putative effects when comparing observational and intervention studies, especially when using pure compounds. These discrepancies may in part be explained by differences in intake levels and their bioavailability. Prior to exerting their bioactivity, these compounds must be made bioavailable, and considerable differences may arise due to their matrix release, changes during digestion, uptake, metabolism, and biodistribution, even before considering dose- and host-related factors. Though many insights have been gained on factors affecting secondary plant metabolite bioavailability, many gaps still exist in our knowledge. In this position paper, we highlight several major gaps in our understanding of phytochemical bioavailability, including effects of food processing, changes during digestion, involvement of cellular transporters in influx/efflux through the gastrointestinal epithelium, changes during colonic fermentation, and their phase I and phase II metabolism following absorption.