I have seen many people asking how to configure SharePoint email settings , in this article I will try to explain Ins and Outs of configuring email in share point.
First we will discuss DNS and Email server setup theory:
Email server exist to send and receive email , there is two services/Protocols that are the core of mailing system POP3 and SMTP.
POP3 is used to receive emails
SMTP is used to send emails
nowadays a new service have been developed that relay on Http protocol such as the Hotmail connector or the MS exchange outlook anywhere services , this new services has been developed to solve the problem that some people faced with their ISP blocking SMTP and/or POP3 protocols due to spams , SharePoint can’t use the HTTP service so we will not discuss this service in this article.
we will build a simple environment that contains 4 machines to demonstrate the configuration. I suppose that you know how to install Active Directory and join a machine to the domain , I will cover the DNS configuration , Configuring SMTP server for SharePoint and Hmail Server configuration.
Back to the POP3/SMTP protocols , email servers use SMTP to send emails to other email servers , but how the email server know about other email servers, Email servers depend on DNS through inquiring DNS server for MX record to determine the IP address of the Email server they are sending email to.
Lets walk through example to see How to configure DNS for mail servers , Suppose that your are building SharePoint testing environment that has email servers for Jitpros.local. based on the following Diagram:

JDC01 server 10.83.15.40 has Active Directory and DNS server
SQLSRV Server 10.83.15.41 has hmail & SQL server installed
SPFound Server 10.83.15.43 has SharePoint installed and SMTP server feature
PC1 Workstation 10.83.15.44 has any client with Email client software
All machine will have their DNS server set to The AD server IP 10.83.15.40
Based on the Above Diagram you will end up with Two SMTP servers one on the Hmail server , that will be used by client to send and receive emails , and anther one used by Share Point to receive email. the catch with SharePoint that it does not have a client software to download email form mail server and also it can’t authenticate to email servers , (Note: Most email servers will deny sending emails if the client can’t authenticate with it , but we will allow SharePoint to send emails by configure email relay, you can also configure windows SMTP services to authenticate when sending to specific domains using SMTP remote Domain settings in IIS manager).
SharePoint use a local drop folder to pickup and receive emails , that’s why you need to install SMTP on it.To configure SharePoint to receive email you have two option.
1.Configure a separated Domain or subdomain for SharePoint Sever for example you could create a Subdomain e.g. Sp.jitpros.local , so that all emails that are sent to SharePoint will have the subdomain suffix sp.jitpros.local , when you enable a SharePoint list to receive email it’s address will be listname@sp.jitpros.local. The nice thing about this configuration that you don’t need create a forward rule for each mail list you enable at SharePoint in your email server.
2. The second Option is to keep your SharePoint email domain in the same root domain e.g Jitpros.local and to use Hmail server or any other email server and create a rule that will forward email that are sent to SharePoint lists to the SharePoint local SMTP server , which means each list that gets email enabled at SharePoint should have a corresponding rule at Hmail server.
lets first configure hmail server DNS record:
For email server to function correctly it needs minimum two records , an A record and MX record , you can also use a CA (conical name) record which simple and Alias to the A record or a second name to the same device.the configuration will start on the DNS server as below:
1. On your DNS server Open the DNS manager , if your mail server is part of Domain and has dynamic DNS registration enabled you will find a record to your hmail server already registered in DNS and has an A record SQLSRV 10.83.15.41

2. As a best practice it’s better to create a Conical Name (virtual name) for your email server, this will help you have constant mail server name in case you needed to change/move your physical email server to antehr machine then you will not be stuck with the old physical machine name. it has been a common practice to set two names for your email server one for POP3 services and one SMTP services in case you needed to separate these two roles into different machines but in our case the two services are provide by the same machine so will create two Conical Records that point to the same machine.
3. So while your on the DNS manager Console select the Zone of your Domain e.g. jitpros.local , right click on the zone name and select new Alias (CNAME).

4. In the New Resource Record screen type SMTP in the Alias Name field and click Browse button , double click your DNS server name , double click forward lookup zones , Double click your Zone name (jitpros.local) , scroll down and select your physical mail server name e.g. SQLSRV and Click Ok.

5. Repeat step 4 but this time type POP in the Alias Name Field , this will create two records pop.jitpros,local and smtp.jitpros.local that point to the same physical server sqlsrv. you can check this by pinging these names they should replay with same IP 10.83.15.41
6. After configuring the CNames record , we need to create an MX record that actually inform other email servers where to deliver email that are sent to your Domain, to create MX record right click the zone name and select New Mail exchange (MX) record

7.In the New Resource record screen , Keep the Host or Child domain filed empty ( which means any email sent to your default domain name ) in the Fully Qualified Domain Name (FQDN) of mail server record type: smtp.jitpros.local (or press browse , double click your DNS server name , double click forward lookup zones , Double click your Zone name (jitpros.local) , select all records form the record type drop box , scroll down and select your SMTP Conical name record and Click Ok) , then Click OK.

what we have done here is that we informed our DNS server that if some one will be sending email to any user at the Domain jitpros.local , the email should be delivered to the sever that is named smtp.jitpros.local.
to test your configuration open a command prompt ant type:
c:\> nslookup
Default Server: jdc01.jitpros.local
Address: 10.83.15.40
> set type=MX
> jitpros.local
Server: jdc01.jitpros.local
Address: 10.83.15.40
jitpros.local MX preference = 10, mail exchanger = smtp.jitpros.local
sqlsrv.jitpros.local internet address = 10.83.15.41
8. Now it’s time to configure your Hmail Server settings , logon to the Hmail server at right click Domains select add , type in your Domain Name e.g. jitpros.local , check the enabled box
9. under your new domain select Add AD account , select your domain name , and select the users account that you want to create email account from them.
10. under setting—> Protocols –> SMTP , select the FRC complaint Tab make sure you select Allow plain authentication .

11. Most email server require authentication to allow users to send email , you can configure SMTP authentication under settings –>Advanced –>IP Range –> require SMTP Authentication –> internet, under require SMTP authentication select when the mail client should authencate to send email , by selecting any of the option below. if you require that all user should authenticate before sending email (this is done so that no one can use your server without authentication to send spam emails ) , select all check boxes.

Your SharePoint server can send emails either through it’s local SMTP server or through Hmail server , but since SharePoint server can’t authenticate to SMTP server , you need to allow it to send email through Hmail server without authentication if you decided to send all email through Hmail server , To configure Hmail server to allow SharePoint server to send emails without authentication , select IP Ranges –> click add –> type in a name for you SharePoint server , assign it a priority that is higher than the other existing rages , clear the require authentication check box for sending option that you need then click Save.

12. You can now configure outlook or any email client to send and received email through this email server , notice that if you configured require SMTP authentication , you client should be configured to use SMTP authentication.
13. you can test your email serve by sending and receiving emails.