Troubleshooting the Inbound Email Scheduler

    Overview

    The Check Inbound Mailboxes scheduler automatically retrieves unread emails from the system inbound email accounts configured in Admin > Inbound Email. Occasionally, the scheduler may fail or be delayed when attempting to import the emails. This article covers troubleshooting tips based on some common causes. 

    Prerequisites

    Some of the following troubleshooting steps require direct access to the database server or the inbound email server. To complete these steps, you must host Sugar on-site or enlist the help of a certified Sugar partner. At a minimum, you must log into Sugar as an admin user to access the schedulers and inbound email settings in the Admin panel.

    For more information on email configuration and schedulers, please refer to the Email and Schedulers documentation.

    Symptoms

    When the Check Inbound Mailboxes scheduler has encountered a problem, you may experience one or more of the following symptoms:

    • Some or all of the email account’s new messages fail to be imported to Sugar.
    • There are excessive delays when sending or receiving messages.
    • You see an “Invalid Inbound Email Credentials” error message.
    • The Check Inbound Mailboxes scheduler job is failing.

    Resolution

    Inbound emails are imported through the Check Inbound Mailboxes scheduler job. When this job executes, it cycles through all of the active inboxes specified in Admin > Inbound Email. The import process pulls any unread messages from the inbox into Sugar. Once imported, the emails are marked as “read” in the inbox. The following sections explain some common reasons that the inbound email scheduler may not complete these steps as expected.

    • Confirm the Connection to the Server
    • Verify the IMAP Connection
    • Verify the Inbound Port Numbers
    • Resolve an “Invalid Inbound Email Credentials” Error
    • Confirm the Unread Status of Missing Messages
    • Check for Duplicate Messages
    • Isolate a Problematic Email
    • Adjust the Timeout Threshold

    Confirm the Connection to the Server

    If Sugar cannot connect to the mail server specified in Admin > Inbound Email, it may prevent you from saving the inbound email settings. Evaluate the following external factors that may be blocking the connection:

    • Firewalls or other network infrastructures
    • Incorrectly compiled PHP
    • Failing SSL connection 

    If your organization’s network administrator has confirmed that these items are properly configured and you still cannot save the inbound email settings, try switching to a POP3 server option if it is available.

    Verify the IMAP Connection

    When polling email accounts, if Sugar cannot make an IMAP connection, it will be reflected in the Sugar Log. On-site customers can set the system’s logging level to “Debug” via Admin > System Settings > Log Level. SugarCloud customers should refer to the Logging section of the SugarCloud Policy Guide for more information about log levels.

    The Sugar Log will display output similar to the following line when the IMAP connection fails:

    [FATAL] SCHEDULERS: could not get an IMAP connection resource for ID [ 139fb6c5-a66c-00fa-0645-507f55f9eb6a ]. Skipping mailbox [ My Mailbox Name ].
    

    In most cases, this is nothing to worry about. If this message appears frequently, you should verify the connection to your mail account with Telnet or OpenSSL. For more information, please refer to the Knowledge Base article Testing Outbound Email Using Command Line. 

    Verify the Inbound Port Numbers

    You may experience issues if the port numbers specified in the Inbound Email settings are incorrect. Navigate to Admin > Inbound Email and open the affected inbound email account. Verify the port numbers according to the following tables, which list some common email and web port numbers.

    Common Email Port Numbers

    Email Servers Port Numbers
    POP3 110
    POP3 – SSL 995
    IMAP 143
    IMAP – SSL 993
    SMTP 25
    SMTP Alternate 26 or 587
    SMTP – SSL 465

    Common Web Port Numbers

    Web Servers Port Numbers
    HTTP 80
    SSL 443
    FTP 21
    FTPs 990
    SFTP 22
    SFTP Shared/Reseller Servers 2222
    Webdisk 2077
    Webdisk – SSL 2078
    MySQL 3306
    MSSQL 1433
    SSH 22
    SSH Shared/Reseller Servers 2222

    Resolve an “Invalid Inbound Email Credentials” Error

    If you see an error stating that the credentials for the affected inbound email account are not valid, navigate to Admin > Inbound Email, open the mail account, and re-enter the inbound email credentials to confirm that they are correct. Once saved, navigate to Admin > Repair and click “Rebuild Relationships”. This will rebuild the relationship metadata and drop the cache file in to confirm that your changes are saved and processed.

    Confirm the Unread Status of Missing Messages

    Sugar will not process emails that are marked as “read” on the server. If some email messages are coming through to Sugar but others are not, confirm that the missing email messages are in an unread state.

    Check for Duplicate Messages

    If you have confirmed that a missing message is marked as unread but it is still not importing to Sugar, the message may have been identified as a duplicate. The next two sections explain how to find out if the message is a duplicate of another message in Sugar. 

    Checking the Database

    If you are not sure whether the email has imported to Sugar, you can check the database for inbound emails sent to or from a specific email address by using the following query, replacing <email address> with the relevant value. 

    SELECT emails.id, emails.name, emails_text.description, emails_text.to_addrs, emails_text.from_addr, emails.date_sent, emails.date_entered 
    FROM emails_text
    INNER JOIN emails ON emails.id = emails_text.email_id
    INNER JOIN emails_email_addr_rel ON emails_email_addr_rel.email_id = emails.id
    INNER JOIN email_addresses ON email_addresses.id = emails_email_addr_rel.email_address_id
    WHERE type = 'inbound' AND email_addresses.email_address = '<email address>'
    ORDER BY date_sent DESC

    To limit the results to emails imported in the last 7 days, adjust the WHERE statement as in the following query, again replacing <email address> with the relevant value:

    SELECT emails.id, emails.name, emails_text.description, emails_text.to_addrs, emails_text.from_addr, emails.date_sent, emails.date_entered 
    FROM emails_text
    INNER JOIN emails ON emails.id = emails_text.email_id
    INNER JOIN emails_email_addr_rel ON emails_email_addr_rel.email_id = emails.id
    INNER JOIN email_addresses ON email_addresses.id = emails_email_addr_rel.email_address_id
    WHERE date_sent > DATE_SUB(NOW(), INTERVAL 7 DAY) AND type = 'inbound' AND email_addresses.email_address = '<email address>'
    ORDER BY date_sent DESC

    Checking the Logs

    When a new email is importing, Sugar’s debug log will contain some key notes about the import. On-site customers can set the system’s logging level to “Debug” via Admin > System Settings > Log Level. SugarCloud customers should refer to the Logging section of the SugarCloud Policy Guide for more information about log levels.

    The following sample content from a log file shows an email importing to Sugar:

    [DEBUG] InboundEmail processing 1 email 24-------------------------------------------------------------------------------------- [DEBUG] *********** InboundEmail doing dupe check.
    [DEBUG] ********* InboundEmail found [ inbox@sugar.crm ] as the destination address for email [ <CE8D775B.5AF08%inbox@sugar.crm> ]
    ...
    [DEBUG] *********** NO duplicate found, continuing with processing.
    ...
    [DEBUG] InboundEmail found multipart email - saving attachments if found.
    [DEBUG] Localization: translating [<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); "><div><div><div style="font-family: Calibri, sans-serif;  ">Heres my example email</div></div></div></body></html>  ] from us-ascii into UTF-8
    [DEBUG] Localization: translating [Here's my email body] from us-ascii into UTF-8
    ...
    [DEBUG] ********************************* InboundEmail finished import of 1 email: Example Email
    

    If the email already exists in Sugar, the log file will contain a section similar to the following:

    [DEBUG] InboundEmail processing 1 email 24-------------------------------------------------------------------------------------- [DEBUG] *********** InboundEmail doing dupe check.
    [DEBUG] ********* InboundEmail found [  inbox@sugar.crm  ] as the destination address for email [ <CE8D775B.5AF08%inbox@sugar.crm> ]
    ...
    [DEBUG] InboundEmail found a duplicate email with ID (36ef71c4474ff9f07d64f3999d0df432)
    [INFO] InboundEmail found a duplicate email: <CE8D775B.5AF08%inbox@sugar.crm>
    

    The log tells you the ID of the duplicate email record in Sugar, so you can now find the original message in Sugar or delete it from the database as needed.

    Isolate a Problematic Email

    Sometimes the import process may be hanging on a specific email. When this happens, you will notice that the scheduler job is failing and that the latest emails have not been marked as read or imported into Sugar. To keep your instance moving and importing records, the best approach is to identify the problematic email and mark it as read. The sections below will outline the various ways to identify the email. Once you are up and running, you should then file a support case so that we can further investigate the issue.

    Fetching the Problematic Email by Message Number

    If you find that the system is hanging, you can fetch the problematic email by its message number. To do this, on-site customers can set the system’s logging level to “Debug” via Admin > System Settings > Log Level. SugarCloud customers should refer to the Logging section of the SugarCloud Policy Guide for more information about log levels.

    Once the scheduler runs and its attempt to pull the email fails, you will see a line or lines similar to the following in the log:

      [DEBUG] InboundEmail processing 1 email 24-----------------------------------------------------------------------------------------
    

    The log will display this text for each message it analyzes. Find the last occurrence of this message in the log. If the scheduler is hanging on an email, it will hang on the last email it tried to import. In this case, “24” is the number of the message that has stalled the process. If you cannot identify the number of the problematic email message, please move on to the next section, Fetching the Problematic Email by Brute Force.

    Next, gather the values that correspond to each component described in the following table. These values will be used to compose a connection string in later steps.

    Name Description Example Value
    Server URL The mail server’s URL imap.gmail.com
    Port The mail server’s port 993
    Protocol The mail server’s protocol imap
    Service The service string determined by the database’s inbound_email.service field for your inbound email account’s database record

    • To find this value, retrieve the field value in the inbound_email.service field. Remove the protocol designation (e.g. “pop3” or “imap”) and extract the remaining text elements to get the service string. If more than one element exists for the string, separate them with a forward slash.
    • Example A : If the database value for this field is “::::ssl::pop3::::::”, omit “pop3” and use the remaining string “ssl” as the Service element of the connection string.
    • Example B : If the database value for this field is “::::::imap::novalidate-cert::notls::”, omit “imap” and use the remaining text (imap, novalidate-cert, notls)” as the Service element of the connection string, placing a forward slash between each text item to get “novalidate-cert/notls”.
    novalidate-cert/notls
    Mailbox The name of the mailbox folder that contains the messages you want Sugar to retrieve INBOX

    To compose the connection string for your instance, arrange the values that correspond to this table in the following format: 
    {Server URL:Port/service=Protocol/Service}Mailbox

    Using the example values in the table, the connection string will be {imap.gmail.com:993/service=imap/ssl}INBOX.

    Alternatively, you can identify your system’s connection string by modifying the ./inbound/modules/InboundEmail/InboundEmail.php file. Loate the following line:

    $connectString = $this->getConnectString($service, $this->mailbox);

    Immediately below this line, add a new line with the following code:

    $GLOBALS['log']->debug("Using connection string: {$connectString}");

    This will log the connection string to the log file so that you can retrieve it there, immediately following the text “Using connection string:”. Copy the connection string and then identify the problematic email using the following script, replacing <your connection string><username>, and <password> with the actual values.

    <?php
    //mailbox to connect to
    $connectionString = '<your connection string>';
    //the message number to fetch
    $messageNumber = 24;
    //open imap connection
    $imap = imap_open($connectionString, '<username>', '<password>');
    echo '<pre>';
    echo '<p>-- Status information about the given mailbox --<p>';
    print_r(imap_status($imap, $mbox, SA_ALL));
    echo '<p>-- Information about the given message --<p>';
    print_r(imap_headerinfo($imap, $messageNumber ));
    echo '<p>-- The header of the specified message --<p>';
    print_r(imap_fetchheader($imap, $messageNumber ));
    echo '</pre>';

    Once you have identified the problem email message, log in to your inbox and mark it as “read” so that the import process skips the email and continues importing the next email.

    Fetching the Problematic Email by Brute Force

    If you do not know the problem message’s number, open the inbox and change the oldest unread email in the monitored folder to “read”. Cycle the scheduler. You may need to repeat these two steps multiple times until the import process skips the current email and continues to the next email. This means that you have found the problem email and resolved the stalled import.

    Adjust the Timeout Threshold

    Depending on your symptoms, you may want to increase or decrease the job timeout configuration. By default, Sugar forces a scheduler job to fail If it attempts to run longer than the threshold. It then batches the remaining job for the next time the scheduler runs. Specified in seconds, the default jobs timeout threshold configuration is 3600 (e.g. 1 hour). For more information on job queues, please refer to the Scheduler Jobs documentation in the Developer Guide.

    If you are a larger organization that is regularly importing a large number of emails but find that there is a delay when they are coming in, try increasing the timeout threshold by adding the following line to config_override.php

      $sugar_config['jobs']['timeout'] = 7200;
    

    Setting the limit to 7200 doubles the default timeout to two hours. You can incrementally add more time to troubleshoot your issue, but it is not recommended to exceed a timeout configuration of 14400 (e.g. 4 hours) because increasing the timeout limit too much could negatively affect system performance.

    Alternatively, importing your inbound email may cause unwanted stress on other parts of the system, which could result in other jobs not running as often as desired. You can attempt to expedite the import by decreasing the timeout threshold. To decrease the limit, add the following line to config_override.php

      $sugar_config['jobs']['timeout'] = 1800;
    

    Setting the limit to 1800 decreases the default timeout by half to 30 minutes. This allows other jobs to continue without waiting for the slow job to complete. For the Inbound Email Scheduler, Sugar will process the imported emails in batches every time the job runs, reducing strain on the server and freeing up resources for other queued jobs to run.

    in Schedulers

    Reach out to us for help