text representation does not impose any restrictions on the column values. Currently on our instances we have them stored both with and without "0x" prefixes, for example. Some non-byte values can also be stored there by mistake. Changing the type to bytea will implicitly allows to store only real byte values in the column.

Besides, bytea values will require twice less storage in comparison with hex encoded text.