XCOM supports Binary, ASCII and EBCDIC format files.
There is no limit to the size of the file.
All platforms support binary. You should always transfer image files as binary. When XCOM transfers a binary file, it transfers it without modifying it in any way.
On XCOM for Linux, Unix and Windows, you would use the parameter code_flag set to binary and carriage_flag=no.