Send VPC flow logs to Sumologic

VPC flow is a AWS feature that allows you to capture the inbound and outbound traffics of all network interfaces in your VPC. To get the information,  you have to go to the Cloudwatch to read the raw logs line by line which is not really efficient. If you happen to use Sumologic, you can send VPC flow logs to it, then define some queries to create a nice dashboard.

This the one that I setup for my VPC. It shows the top 10 source ip by hits or size, top 10 destination ip by hits or size, top accepted or rejected ip…

 
Here are the how-to steps:

1. Enable VPC flow, please refer AWS doc.
2. Setup a collector in your VPC to get the logs then send to Sumologic, please refer Sumologic doc.
3. Setup the dashboard, here are some sample of the queries:

Top 10 source ip by hits (excluding local ip e.g 172.31.*)

_sourceCategory=vpc <your_vpc_log_group>
| json "message","logStream","logGroup"
| parse field=message "* * * * * * * * * * * * * *" as version,accountID,interfaceID,src_ip,dest_ip,src_port,dest_port,Protocol,Packets,bytes,StartSample,EndSample,Action,status
| where !(src_ip matches "172.31.*")
| count as Records by src_ip | top 10 src_ip by Records

Top 10 accepted source ip from external that hit network interface eni-1234abcd or eni-5678efgh

_sourceCategory=vpc <your_vpc_log_group> ACCEPT
| json "message","logStream","logGroup"
| parse field=message "* * * * * * * * * * * * * *" as version,accountID,interfaceID,src_ip,dest_ip,src_port,dest_port,Protocol,Packets,bytes,StartSample,EndSample,Action,status
| where !(src_ip matches "172.31.*") and interfaceid in ("eni-1234abcd", "eni-5678efgh")
| count as Records by src_ip | top 20 src_ip by Records

Map

_sourceCategory=vpc <your_vpc_log_group>
| json "message","logStream","logGroup"
| parse field=message "* * * * * * * * * * * * * *" as version,accountID,interfaceID,src_ip,dest_ip,src_port,dest_port,Protocol,Packets,bytes,StartSample,EndSample,Action,status
| where !(src_ip matches "172.31.*") and interfaceid in ("eni-1234abcd", "eni-5678efgh")
| lookup latitude, longitude, country_code, country_name, region, city, postal_code, area_code, metro_code from geo://default on ip = src_ip
| count by latitude, longitude, country_code, country_name, region, city, postal_code, area_code, metro_code

*Note*: The private IP address of ELB can not be found in the AWS console, you have to run CLI to get it. Here is the command:

aws ec2 describe-network-interfaces --filters "Name=description,Values=ELB*" |  jq -r '.NetworkInterfaces[] | "NAME: \(.Description) INTERFACE: \(.NetworkInterfaceId) IP: \(.PrivateIpAddress)"'