Since its inception, the north star of artificial intelligence was to design systems capable of learning as efficiently (i.e. with limited training signal) and effectively (i.e. demonstrating good performances) as humans to solve challenging problems that require human-like intelligence. Deep neural networks and the collection of popular deep learning ingredients used to produce systems usable in the real world, such as optimization algorithms, novel architectures, objective functions, and large annotated datasets, have shown remarkable performances across various tasks in recent years. However, this dominant paradigm requires a large amount of fully labeled data, which is often expensive and difficult to acquire. It might also contain annotation errors and hidden biases, which limits the applicability and adoption of such systems. Yet humans demonstrate a remarkable ability to learn effectively across diverse settings, using limited supervision and leveraging prior experience to adapt to novel cases and gain new skills quickly. This discrepancy raises an obvious question, can we design systems with similar capabilities? In this thesis, we aim to develop label-efficient learning algorithms that are effective with a limited or no amount of annotated examples for various tasks over different modalities and multiple levels of abstraction. To this end, this thesis cover works that: i) develop learning methods for paradigms with varying degrees of supervision, ii) present results for different modalities, notably vision, and text, and iii) different tasks across various levels of abstraction (e.g., image level and pixel level). We hope these works can help further advance the state of the field and aid in developing systems capable of learning efficiently and adapting effectively across a wide range of environments.