It is a myopic Western view, which knows Yoga as a physical exercise, to claim that is all that Yoga is. The physical is only one of the eight limbs of the classic form of Yoga known as Ashtanga Yoga:
Yama - ethical practices and restraints
Niyama - religious observances, commitments to practice, such as study and devotion
Asana - integration of mind and body through physical postures
Pranayama - breath control
Pratyahara - withdrawal of external sensory input
Dharana - concentration, one-pointedness of mind
Dhyana - meditation (quiet activity that leads to samadhi)
Samadhi - the quiet state of blissful awareness, ecstasy, dissolution of separateness
If we relate to Yoga merely as a mat practice of postures, as a tool disconnected from its history, then it is not religion, but neither is it Yoga. It is more like a bio-feedback or a stress reduction method that benefits our body-mind. Calling an isolated set of practices “Yoga” does not change that Yoga is a comprehensive tradition, a religion reflecting the collective wisdom of thousands of years of devotion by its adherents.
Yoga is a body of knowledge, a wisdom tradition that is deep and vast. It deserves our treating it as such, and, in its full expression, a religion.