Agile Central allows including attachments (see link for more info) to many objects and artifacts.
Is there an impact on performance if we'll have many attachments of the max size (50MB), even all attached to the same artifact?
The short answer is: "No. There is no impact on performance if you have even many file attachments, even to the same object/artifact, even if all are of the max 50MB file size".
The reason lies with how the Attachment object is modeled. This is described in our WSAPI Object Model.
The Attachment object stores the description of the attachment, its name, size, the user who added it, the artifact it belongs to. Yet, it does not contain the actual blob of the file itself. The data blob is stored in another object: "AttachmentContent". The Attachment object also keeps a one-to-one reference to its AttachmentContent.
What this means is that when Agile Central or any of its apps are looking to display whether or not artifacts include attachments, their count, even names or description, even the user who added them, or even the content type - they will not have to query for the actual blob of the file. The data that's kept in the Attachment object is not at all large. Hence the performance will be normal and not be impacted even if many large blobs belong to any artifact or number of artifacts.
The only time the blob is accessed will be when a user is asking to view or download the file.
As a reference, see some of the details of the Attachment object followed by the AttachmentContent object: