Maileater does not properly parse HTML emails from Gmail based clients

Document ID : KB000122927
Last Modified Date : 17/12/2018
Show Technical Document Details
Introduction:
With CA Service Desk Manager 17.1 release onwards, the maileater program supports HTML/Rich Text Format (RTF) emails. This new maileater functionality works fine most of the time. Even if the mail is sent to maileater in RTF/HTML format, both Plain Text and HTML message bodies are received by the mailbox. Such emails are processed fine, including parsing of Service Desk attribute/value pairs. 

Example, if there are attribute value pairs like below with in the body of an email that the maileater receives, the maileater attempts to set Category of Incident #1234 to Email and its status to Closed.
 
%INCIDENT_ID=1234
%STATUS=Closed
%CATEGORY=Email
Question:
Even if the attribute value pairs in the body are formatted using special fonts / sizes / character formats etc., they are normally parsed fine by the maileater.

However, if the same content is received by maileater from a mail sender using Gmail / Google mail based email client, the parsing is not done properly. Below is an example trace output from maileater for a simple with body:


%STATUS=Resolved
%CATEGORY=Email
 
Line one
 
Line two
 
 
 
 

Trace entries from NX_ROOT/log/maileater_nxd.log for the above shows up like:
 
2018-12-12 08:13:45:227 DEBUG  [ForkJoinPool-1-worker-1] c.c.S.m.JavaMailParser - [mailbox:testmail@company.com:100] Start processing the Email message from (first.last@customer.com) of subject (cr)
2018-12-12 08:13:45:227 TRACE  [ForkJoinPool-1-worker-1] c.c.S.m.JavaMailParser - [mailbox:testmail@company.com:100] This is a Multipart
2018-12-12 08:13:45:227 TRACE  [ForkJoinPool-1-worker-1] c.c.S.m.JavaMailParser - ---------------------------
2018-12-12 08:13:45:227 TRACE  [ForkJoinPool-1-worker-1] c.c.S.m.JavaMailParser - This is plain text
2018-12-12 08:13:45:227 TRACE  [ForkJoinPool-1-worker-1] c.c.S.m.JavaMailParser - ---------------------------
2018-12-12 08:13:45:227 TRACE  [ForkJoinPool-1-worker-1] c.c.S.m.JavaMailParser - [mailbox:testmail@company.com:100]  cr

*%STATUS=Resolved*

*%CATEGORY=Email*

*Line one*

*Line two*

2018-12-12 08:13:45:227 TRACE  [ForkJoinPool-1-worker-1] c.c.S.m.JavaMailParser - This is html text
2018-12-12 08:13:45:227 TRACE  [ForkJoinPool-1-worker-1] c.c.S.m.JavaMailParser - ---------------------------
2018-12-12 08:13:45:227 TRACE  [ForkJoinPool-1-worker-1] c.c.S.m.JavaMailParser - <html><head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="MS Exchange Server version rmj.rmm.rup.rpr">
<title>cr</title>
</head>
<body>


<p dir="LTR"><span lang="en-us"><b><font face="Calibri">%STATUS=Resolved</font></b></span></p>

<p dir="LTR"><span lang="en-us"><b><font face="Calibri">%CATEGORY=Email</font></b></span></p>

<p dir="LTR"><span lang="en-us"><b><font face="Calibri">Line one</font></b></span></p>

<p dir="LTR"><span lang="en-us"><b><font face="Calibri">Line two</font></b></span></p>

<p dir="LTR"><span lang="en-us"><b></b></span></p>
<br>

<p dir="LTR"><span lang="en-us"><b><font face="Calibri"></font></b></span><span lang="en-us"><b>&nbsp;</b></span></p>

</body>
</html>

2018-12-12 08:13:45:227 DEBUG  [ForkJoinPool-1-worker-1] c.c.S.m.JavaMailParser - [mailbox:testmail@company.com:100] Attach Entire Email is set to Yes!!!
2018-12-12 08:13:45:227 DEBUG  [ForkJoinPool-1-worker-1] c.c.S.m.JavaMailParser - [mailbox:testmail@company.com:100] Attaching the received Email message as file
2018-12-12 08:13:45:243 DEBUG  [ForkJoinPool-1-worker-1] c.c.S.m.JavaMailParser - [mailbox:testmail@company.com:100] Start uploading the email as an attachment to repository
2018-12-12 08:13:45:274 DEBUG  [ForkJoinPool-1-worker-1] c.c.S.m.JavaMailParser - [mailbox:testmail@company.com:100] Uploaded the email attachment successfully to the repository
2018-12-12 08:13:45:274 DEBUG  [ForkJoinPool-1-worker-1] c.c.S.m.JavaMailParser - [mailbox:testmail@company.com:100] Processing the Email From(first.last@customer.com) Subject(cr) Body( cr

*%STATUS=Resolved*

*%CATEGORY=Email*

*Line one*

*Line two*

%ATTACHMENT={{5C65AD8E6B6FE347A5827F784AFD6B51_IMAP6923711092002389874.eml}}Email_12122018_081345.eml
%RELPATH={{rep_folder_00005}}rep_folder_00005


%DOCREP={{Service Desk}}doc_rep:1002
)




Notice that, the formatting is all messed up in the Plain Text message that's parsed by maileater.  Every formatted string has a preceding and trailing *  characters.
Environment:
Service Desk Manager 17.1 / 17.1.x
Answer:
Gmail formats HTML content slightly different than other email clients do. It adds additional meta characters (example: *text*  for text ).  These characters are present even in the plain-text version of the formatted email content. Other email clients (example: Microsoft Outlook) does not insert any such characters on the plain-text version of such emails. 

Maileater basically looks at the plain-text version of the emails to process the content and does not perform any additional parsing on the content. At this time maileater works under the assumption that the plain-text version does not have any additional meta characters like the ones above.

Because of the same, maileater now will not be able to parse the attribute value pairs properly. So, %STATUS=CLOSED  as its parsed as *%STATUS=CLOSED* .

As *%STATUS  is not an SDM attribute, its treated as if its just text that needs to go to description of an activity log comment or something similar.  


So, it is our recommendation at this time that, if there are any such attribute/value pairs that need to be sent to maileater, that those be sent in un-formatted text to keep consistency between different email clients. 

SDM email notifications being sent out (to which customers reply, to update tickets etc.,) should not have formatted text for the SDM attribute value pairs because of the above findings.