The assumption of Lorentz invariance is one of the founding principles of modern physics and violation of that would have deep consequences to our understanding of the universe. Potential signatures of such a violation could range from energy dependent dispersion introduced into a light curve to a change in the photon-photon pair production threshold that changes the expected opacity of the universe. Astronomical sources of Very High Energy (VHE) photons can be used as test beams to probe fundamental physics phenomena, however, such effects would likely be small and need to be disentangled from intrinsic source physics processes. The Cherenkov Telescope Array (CTA) will be the next generation ground based observatory of VHE photons. It will have improved flux sensitivity, a lower energy threshold (tens of GeV), broader energy coverage (nearly 5 decades) and improved energy resolution (better than 10\% over much of the energy range) compared to current facilities in addition to excellent time resolution for short timescale and rapidly varying phenomena. The expected sensitivity of this facility leads to us to examine in this contribution the kinds of limits to Lorentz Invariance Violation (LIV) that we could expect to obtain on VHE observations of Active Galactic Nuclei (AGN), Gamma Ray Bursts (GRBs) and pulsars with CTA. With a statistical sample and wide variety of sources CTA has the potential to set model independent limits.