Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 3 Next »

This page contains the Hostname Field Extraction methods for a number of Log Sources, listed by log source.

The Armor Log Relay allows customers to forward security logs to Armor from a multitude of available Armor supported log sources. Many of these Log Relay log sources may be forwarding events from multiple event sources and it is important to be able to uniquely identify the originating event source. This facilitates easy searching with Kibana, tagging in Armor's Management Portal (AMP), and enhances the security outcomes provided by Armor.

Armor extracts a hostname from each log event as the unique identifier of the event's source. In Kibana, that hostname is mapped to the logsource.hostname field and in the Armor Tags API, it is combined in the resourceId for each tag ID using the convention log-relay-core-instance-id::hostname. Each log source has a specific way of formatting their logs and below is a description of the methodology Armor uses to extract the hostname value from each device type.

Check Point

Armor extracts the hostname by evaluating the following possible field matches, in order:

Extraction method 1:

The extraction is done by a key-value pair match on the origin field.

Sample Log

LEEF:2.0|Check Point|VPN-1 & FireWall-1|1.0|Accept|cat=VPN-1 & FireWall-1 devTime=1580842617 srcPort=64416 layer_name=Rulebase Network layer_name=App Control - URL Filter layer_uuid=0c0c8c0c-1b7f-49b5-9b74-0843569dec02 layer_uuid=51f33b96-a02f-4666-9ae7-644d2ca7e116 match_id=184 match_id=16777227 parent_rule=0 parent_rule=0 rule_action=Accept rule_action=Accept rule_name=LOCK-DOWN rule_name=Cleanup rule rule_uid=d8b4a241-c952-4379-b0d7-c96eec992b46 rule_uid=55c97db7-8303-47ed-bbec-a657de6ed15d action=Accept ifdir=inbound ifname=eth1.1111 logid=0 loguid={0x5e39be79,0x4,0xf13c400a,0xc0000003} origin=127.0.0.1 originsicname=CN\=EALEDIcp5800fwa,O\=EALEDICPMGR..i7ngdz sequencenum=24 version=5 dst=8.8.8.8 inzone=Internal outzone=External proto=6 service=443 service_id=https src=192.168.1.1

Hostname is the origin field, so in this sample, the hostname would be 127.0.0.1.

Base case extraction method:

The expected extraction fields are not in the log, so the fallback option is unknown-check-point-<originating-host>.

Sample Log
LEEF:2.0|Check Point|VPN-1 & FireWall-1|1.0|Accept|cat=VPN-1 & FireWall-1 devTime=1580842617 srcPort=64416 layer_name=Rulebase Network layer_name=App Control - URL Filter layer_uuid=0c0c8c0c-1b7f-49b5-9b74-0843569dec02 layer_uuid=51f33b96-a02f-4666-9ae7-644d2ca7e116 match_id=184 match_id=16777227 parent_rule=0 parent_rule=0 rule_action=Accept rule_action=Accept rule_name=LOCK-DOWN rule_name=Cleanup rule rule_uid=d8b4a241-c952-4379-b0d7-c96eec992b46 rule_uid=55c97db7-8303-47ed-bbec-a657de6ed15d action=Accept ifdir=inbound ifname=eth1.1111 logid=0 loguid={0x5e39be79,0x4,0xf13c400a,0xc0000003} originsicname=CN\=CN_FIELD,O\=O_FIELD..i7ngdz sequencenum=24 version=5 dst=8.8.8.8 inzone=Internal outzone=External proto=6 service=443 service_id=https src=192.168.1.1

If the log originated from 127.0.0.1, then the hostname would be unknown-check-point-127.0.0.1.

Cisco ASA

Armor extracts the hostname by evaluating the following possible field matches, in order:

Extraction Method 1:

The extraction is performed by a grok match on the log as follows: match => {"message" => "<%{NUMBER}>%{CISCOTIMESTAMP} %{WORD:syslog_hostname}"}.

Sample Log

<179>Feb 3 06:54:54 cisco-asa-device mysql_log 2020-02-03T14:54:54.250208Z 0 [Warning] InnoDB: Table mysql/innodb_table_stats has length mismatch in the column name table_name. Please run mysql_upgrade


The hostname is the syslog hostname from the log, so the hostname here would be cisco-asa-device.

Extraction Method 2:

The extraction is performed by a grok match on the log as follows: match => {"message" => "<%{NUMBER}>%{DATA:syslog_hostname} %%{CISCOTAG}:"}.

Sample Log
<132>cisco-asa-device %ASA-4-106023: Deny tcp src outside:8.8.8.8/44761 dst eqf:192.168.1.1/28967 by access-group "outside_access_in" [0x0, 0x0]

The hostname is the Cisco header hostname field from the log, so the hostname here would be cisco-asa-device.

Base case extraction method:

The expected fields are not in the log, so the fallback option is unknown-cisco-asa-<originating-host>.

Sample Log
<132>%ASA-4-106023: Deny tcp src outside:8.8.8.8/44761 dst eqf:192.168.1.1/28967 by access-group "outside_access_in" [0x0, 0x0]

If the log originated from the host 127.0.0.1, then the hostname would be unknown-cisco-asa-127.0.0.1.

Cisco ISR

Armor extracts the hostname by evaluating the following possible field matches, in order:

Extraction Method 1:

The extraction is performed by a grok match on the log as follows: match => {"message" => "<%{NUMBER}>%{CISCOTIMESTAMP} %{WORD:syslog_hostname}"}.

Sample Log

<179>Feb 3 06:54:54 cisco-isr-device mysql_log 2020-02-03T14:54:54.250208Z 0 [Warning] InnoDB: Table mysql/innodb_table_stats has length mismatch in the column name table_name. Please run mysql_upgrade


The hostname is the syslog hostname from the log, so the hostname here would be cisco-isr-device.

Extraction Method 2:

The extraction is performed by a grok match on the log as follows: match => {"message" => "<%{NUMBER}>%{DATA:syslog_hostname} %%{CISCOTAG}:"}.

Sample Log
<132>cisco-isr-device %ASA-4-106023: Deny tcp src outside:8.8.8.8/44761 dst eqf:192.168.1.1/28967 by access-group "outside_access_in" [0x0, 0x0]

The hostname is the Cisco header hostname field from the log, so the hostname here would be cisco-isr-device.

Base case extraction method:

The expected fields are not in the log, so the fallback option is unknown-cisco-isr-<originating-host>.

Sample Log
<132>%ASA-4-106023: Deny tcp src outside:8.8.8.8/44761 dst eqf:192.168.1.1/28967 by access-group "outside_access_in" [0x0, 0x0]

If the log originated from the host 127.0.0.1, then the hostname would be unknown-cisco-isr-127.0.0.1.

Fortinet Fortigate

Armor extracts the hostname by evaluating the following possible field matches, in order:

Extraction Method 1:

The extraction is done by a key-value pair match on the devname field.

Sample Log
<189>date=2020-01-30 time=01:34:53 devname="FORT-SAMPLE" devid="ABC1DE2345678901" logid="0000000013" type="traffic" subtype="forward" level="notice" vd="root" eventtime=1580319293 srcip=192.168.1.1 srcport=32471 srcintf="port9" srcintfrole="wan" dstip=8.8.8.8 dstport=443 dstintf="port11" dstintfrole="lan" poluuid="15459248-2e75-51e9-82d7-4fe3d456124a" sessionid=714923758 proto=6 action="server-rst" policyid=121 policytype="policy" service="HTTPS" dstcountry="Canada" srccountry="United States" trandisp="noop" duration=11 sentbyte=6530 rcvdbyte=45758 sentpkt=76 rcvdpkt=46 appcat="unscanned" wanin=43382 wanout=2570 lanin=2570 lanout=2570

Hostname is the devname field, so in this sample, the hostname would be FORT-SAMPLE.

Extraction Method 2:

The extraction is done by a key-value pair match on the devid field.

Sample Log
<189>date=2020-01-30 time=01:34:53 devid="ABC1DE2345678901" logid="0000000013" type="traffic" subtype="forward" level="notice" vd="root" eventtime=1580319293 srcip=192.168.1.1 srcport=32471 srcintf="port9" srcintfrole="wan" dstip=8.8.8.8 dstport=443 dstintf="port11" dstintfrole="lan" poluuid="15459248-2e75-51e9-82d7-4fe3d456124a" sessionid=714923758 proto=6 action="server-rst" policyid=121 policytype="policy" service="HTTPS" dstcountry="Canada" srccountry="United States" trandisp="noop" duration=11 sentbyte=6530 rcvdbyte=45758 sentpkt=76 rcvdpkt=46 appcat="unscanned" wanin=43382 wanout=2570 lanin=2570 lanout=2570

Hostname is the devid field, so in this sample, the hostname would be ABC1DE2345678901.

Base case extraction method:

The expected fields are not in the log, so the fallback option is unknown-fortigate-security-gateway-<originating host>.

Sample Log
<189>date=2020-01-30 time=01:34:53 logid="0000000013" type="traffic" subtype="forward" level="notice" vd="root" eventtime=1580319293 srcip=8.8.8.8 srcport=32471 srcintf="port9" srcintfrole="wan" dstip=8.8.8.8 dstport=443 dstintf="port11" dstintfrole="lan" poluuid="15459248-2e75-51e9-82d7-4fe3d456124a" sessionid=714923758 proto=6 action="server-rst" policyid=121 policytype="policy" service="HTTPS" dstcountry="Canada" srccountry="United States" trandisp="noop" duration=11 sentbyte=6530 rcvdbyte=45758 sentpkt=76 rcvdpkt=46 appcat="unscanned" wanin=43382 wanout=2570 lanin=2570 lanout=2570

If the log originated from the host 127.0.0.1, then the hostname would be unknown-fortigate-security-gateway-127.0.0.1.

Juniper SRX

Armor extracts the hostname by evaluating the following possible field matches, in order:

Extraction Method 1:

The extraction is performed by a grok match on the log as follows: match => {"message" => "%{SYSLOGTIMESTAMP} %{HOSTNAME:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?:"}.

Sample Log
Feb 22 20:35:07 router1 snmpd[359]: SNMPD_THROTTLE_QUEUE_DRAINED: trap_throttle_timer_handler: cleared all throttled traps

The hostname is the syslog hostname from the log, so the hostname here would be router1.

Base case extraction method:

The expected fields are not in the log, so the fallback option is unknown-juniper-srx-<originating host>.

Sample Log

VSRX chassisd 5738 CHASSISD_IFDEV_CREATE_FAILURE [junos@2636.1.1.1.2.129 function-name='create_pics' interface-name='lsq-0/0/0' error-message='Invalid argument'] create_pics: unable to create interface device for lsq-0/0/0 (Invalid argument)


If the log originated from the host 127.0.0.1, then the hostname would be unknown-juniper-srx-127.0.0.1.

Palo Alto PanOS

Armor extracts the hostname by evaluating the following possible field matches, in order:

Extraction method 1:

The extraction is done by a key-value pair match on the DeviceName field.

Sample Log
<14>Dec 12 11:43:52 palodevice LEEF:1.0|Palo Alto Networks|PAN-OS Syslog Integration|8.1.9-h4|allow|cat=TRAFFIC|ReceiveTime=2019/12/12 11:43:52|SerialNumber=123456789123|Type=TRAFFIC|Subtype=start|devTime=Dec 12 2019 16:43:52 GMT|src=192.168.1.3|dst=10.0.2.21|srcPostNAT=8.8.8.8|dstPostNAT=8.8.8.8|RuleName=Inside to Outside|usrName=|SourceUser=|DestinationUser=|Application=paloalto-wildfire-cloud|VirtualSystem=vsys1|SourceZone=inside-zone|DestinationZone=outside-zone|IngressInterface=ethernet1/3|EgressInterface=ethernet1/1|LogForwardingProfile=default|SessionID=57998|RepeatCount=1|srcPort=39936|dstPort=443|srcPostNATPort=34068|dstPostNATPort=443|Flags=0x400000|proto=tcp|action=allow|totalBytes=553|dstBytes=74|srcBytes=479|totalPackets=4|StartTime=2019/12/12 11:43:34|ElapsedTime=0|URLCategory=computer-and-internet-info|sequence=40307562|ActionFlags=0x8000000000000000|SourceLocation=192.168.0.0-192.168.255.255|DestinationLocation=United States|dstPackets=1|srcPackets=3|SessionEndReason=n/a|DeviceGroupHierarchyL1=64|DeviceGroupHierarchyL2=121|DeviceGroupHierarchyL3=144|DeviceGroupHierarchyL4=0|vSrcName=|DeviceName=PANOS-01|ActionSource=from-policy|SrcUUID=|DstUUID=|TunnelID=0|MonitorTag=|ParentSessionID=0|ParentStartTime=|TunnelType=N/A

In this sample log, DeviceName=PANOS-01, so hostname for this log event would be PANOS-01.

Extraction method 2:

The extraction is done by a key-value pair match on the SerialNumber field.

Sample Log
<14>Dec 12 11:43:52 palodevice LEEF:1.0|Palo Alto Networks|PAN-OS Syslog Integration|8.1.9-h4|allow|cat=TRAFFIC|ReceiveTime=2019/12/12 11:43:52|SerialNumber=123456789123|Type=TRAFFIC|Subtype=start|devTime=Dec 12 2019 16:43:52 GMT|src=192.168.1.3|dst=10.0.2.21|srcPostNAT=8.8.8.8|dstPostNAT=8.8.8.8|RuleName=Inside to Outside|usrName=|SourceUser=|DestinationUser=|Application=paloalto-wildfire-cloud|VirtualSystem=vsys1|SourceZone=inside-zone|DestinationZone=outside-zone|IngressInterface=ethernet1/3|EgressInterface=ethernet1/1|LogForwardingProfile=default|SessionID=57998|RepeatCount=1|srcPort=39936|dstPort=443|srcPostNATPort=34068|dstPostNATPort=443|Flags=0x400000|proto=tcp|action=allow|totalBytes=553|dstBytes=74|srcBytes=479|totalPackets=4|StartTime=2019/12/12 11:43:34|ElapsedTime=0|URLCategory=computer-and-internet-info|sequence=40307562|ActionFlags=0x8000000000000000|SourceLocation=192.168.0.0-192.168.255.255|DestinationLocation=United States|dstPackets=1|srcPackets=3|SessionEndReason=n/a|DeviceGroupHierarchyL1=64|DeviceGroupHierarchyL2=121|DeviceGroupHierarchyL3=144|DeviceGroupHierarchyL4=0|vSrcName=|ActionSource=from-policy|SrcUUID=|DstUUID=|TunnelID=0|MonitorTag=|ParentSessionID=0|ParentStartTime=|TunnelType=N/A

In this sample log, SerialNumber=123456789123, so hostname for this log event would be 123456789123.

Extraction method 3:

The extraction is performed by a grok match on the log as follows: match => {"message" => "%{SYSLOGTIMESTAMP} %{SYSLOGHOST:hostname}"}.

Sample Log
<14>Dec 12 11:43:52 palodevice LEEF:1.0|Palo Alto Networks|PAN-OS Syslog Integration|8.1.9-h4|allow|cat=TRAFFIC|ReceiveTime=2019/12/12 11:43:52|Type=TRAFFIC|Subtype=start|devTime=Dec 12 2019 16:43:52 GMT|src=192.168.1.3|dst=10.0.2.21|srcPostNAT=8.8.8.8|dstPostNAT=8.8.8.8|RuleName=Inside to Outside|usrName=|SourceUser=|DestinationUser=|Application=paloalto-wildfire-cloud|VirtualSystem=vsys1|SourceZone=inside-zone|DestinationZone=outside-zone|IngressInterface=ethernet1/3|EgressInterface=ethernet1/1|LogForwardingProfile=default|SessionID=57998|RepeatCount=1|srcPort=39936|dstPort=443|srcPostNATPort=34068|dstPostNATPort=443|Flags=0x400000|proto=tcp|action=allow|totalBytes=553|dstBytes=74|srcBytes=479|totalPackets=4|StartTime=2019/12/12 11:43:34|ElapsedTime=0|URLCategory=computer-and-internet-info|sequence=40307562|ActionFlags=0x8000000000000000|SourceLocation=192.168.0.0-192.168.255.255|DestinationLocation=United States|dstPackets=1|srcPackets=3|SessionEndReason=n/a|DeviceGroupHierarchyL1=64|DeviceGroupHierarchyL2=121|DeviceGroupHierarchyL3=144|DeviceGroupHierarchyL4=0|vSrcName=|ActionSource=from-policy|SrcUUID=|DstUUID=|TunnelID=0|MonitorTag=|ParentSessionID=0|ParentStartTime=|TunnelType=N/A

In this sample log, palodevice is the syslog hostname, so hostname for this log event would be palodevice.

Base case extraction method:

The expected extraction fields are not in the log, so the fallback option is unknown-palo-alto-firewall-<originating-host>.

Sample Log
<14>Dec 12 11:43:52 LEEF:1.0|Palo Alto Networks|PAN-OS Syslog Integration|8.1.9-h4|allow|cat=TRAFFIC|ReceiveTime=2019/12/12 11:43:52|Type=TRAFFIC|Subtype=start|devTime=Dec 12 2019 16:43:52 GMT|src=192.168.1.3|dst=10.0.2.21|srcPostNAT=8.8.8.8|dstPostNAT=8.8.8.8|RuleName=Inside to Outside|usrName=|SourceUser=|DestinationUser=|Application=paloalto-wildfire-cloud|VirtualSystem=vsys1|SourceZone=inside-zone|DestinationZone=outside-zone|IngressInterface=ethernet1/3|EgressInterface=ethernet1/1|LogForwardingProfile=default|SessionID=57998|RepeatCount=1|srcPort=39936|dstPort=443|srcPostNATPort=34068|dstPostNATPort=443|Flags=0x400000|proto=tcp|action=allow|totalBytes=553|dstBytes=74|srcBytes=479|totalPackets=4|StartTime=2019/12/12 11:43:34|ElapsedTime=0|URLCategory=computer-and-internet-info|sequence=40307562|ActionFlags=0x8000000000000000|SourceLocation=192.168.0.0-192.168.255.255|DestinationLocation=United States|dstPackets=1|srcPackets=3|SessionEndReason=n/a|DeviceGroupHierarchyL1=64|DeviceGroupHierarchyL2=121|DeviceGroupHierarchyL3=144|DeviceGroupHierarchyL4=0|vSrcName=|ActionSource=from-policy|SrcUUID=|DstUUID=|TunnelID=0|MonitorTag=|ParentSessionID=0|ParentStartTime=|TunnelType=N/A

If the log originated from 192.168.1.3, then the hostname would be unknown-palo-alto-firewall-192.168.1.3.

Sonicwall

Armor extracts the hostname by evaluating the following possible field matches, in order:

Extraction method 1:

The extraction is done by a key-value pair match on the sn field.

Sample Log

<134> id=firewall sn=1234567891A1 time="2019-11-06 15:22:44" fw=4.4.4.4 pri=6 c=512 m=602 msg="DNS packet allowed" app=2 n=110037 src=8.8.8.8:53:X1 dst=8.8.8.8:36780:X1 srcMac=02:c2:ea:7f:6f:07 dstMac=02:80:85:81:68:12 proto=udp/dns fw_action="forward"


The event's hostname is extracted from the sn field in the log event. In this sample log, sn=1234567891A1, so hostname for this log event would be 1234567891A1.

Extraction method 2:

The extraction is done by a key-value pair match on the fw field.

Sample Log
<134> id=firewall time="2019-11-06 15:22:44" fw=4.4.4.4 pri=6 c=512 m=602 msg="DNS packet allowed" app=2 n=110037 src=8.8.8.8:53:X1 dst=8.8.8.8:36780:X1 srcMac=02:c2:ea:7f:6f:07 dstMac=02:80:85:81:68:12 proto=udp/dns fw_action="forward"

The event's hostname is extracted from the fw field in the log event. In this sample log, fw=4.4.4.4, so hostname for this log event would be 4.4.4.4.

Base case extraction method:

The expected extraction fields are not in the log, so the fallback option is unknown-sonicwall-<originating-host>.

Sample Log
<134> id=firewall time="2019-11-06 15:22:44" pri=6 c=512 m=602 msg="DNS packet allowed" app=2 n=110037 src=4.4.4.4:53:X1 dst=8.8.8.8:36780:X1 srcMac=02:c2:ea:7f:6f:07 dstMac=02:80:85:81:68:12 proto=udp/dns fw_action="forward"

The expected extraction fields are not in the log, so the fallback option is unknown-sonicwall-<originating-host>. If the log originated from 4.4.4.4, then the hostname would be 4.4.4.4.

AWS WAF

Armor extracts the hostname by evaluating the following possible field matches, in order:

Extraction method 1:

The extraction is performed by a grok match on the log as follows: match => {"webaclId" => ".+webacl/(?<hostname>[^/]+)"}.

Sample Log
{"timestamp":1580760260344,"formatVersion":1,"webaclId":"arn:aws:wafv2:us-west-2:123456789012:regional/webacl/sample-web-acl/web-acl-id","terminatingRuleId":"Default_Action","terminatingRuleType":"REGULAR","action":"ALLOW","terminatingRuleMatchDetails":[],"httpSourceName":"ALB","httpSourceId":"123456789012-app/sample-web-acl-alb/ab123cd4ef56789g","ruleGroupList":[{"ruleGroupId":"AWS#AWSManagedRulesAmazonIpReputationList","terminatingRule":null,"nonTerminatingMatchingRules":[],"excludedRules":null}],"rateBasedRuleList":[],"nonTerminatingMatchingRules":[],"httpRequest":{"clientIp":"8.8.8.8","country":"US","headers":[{"name":"Host","value":"sample-web-acl-alb-123456789012.us-west-2.elb.amazonaws.com"},{"name":"User-Agent","value":"curl/7.64.1"},{"name":"Accept","value":"*/*"}],"uri":"/","args":"=php.admin","httpVersion":"HTTP/1.1","httpMethod":"GET","requestId":null}}

The event's hostname is extracted and manipulated from the webaclId field with the following regular expression pattern: .+webacl\/([^\/]+). In this sample log, hostname would be sample-web-acl.

Base case extraction method:

The expected fields are not in the log, so the fallback option is unknown-aws-waf-<originating-host>.

Sample Log
{"timestamp":1580760260344,"formatVersion":1,"terminatingRuleId":"Default_Action","terminatingRuleType":"REGULAR","action":"ALLOW","terminatingRuleMatchDetails":[],"httpSourceName":"ALB","httpSourceId":"123456789012-app/sample-web-acl-alb/ab123cd4ef56789g","ruleGroupList":[{"ruleGroupId":"AWS#AWSManagedRulesAmazonIpReputationList","terminatingRule":null,"nonTerminatingMatchingRules":[],"excludedRules":null}],"rateBasedRuleList":[],"nonTerminatingMatchingRules":[],"httpRequest":{"clientIp":"8.8.8.8","country":"US","headers":[{"name":"Host","value":"sample-web-acl-alb-123456789012.us-west-2.elb.amazonaws.com"},{"name":"User-Agent","value":"curl/7.64.1"},{"name":"Accept","value":"*/*"}],"uri":"/","args":"=php.admin","httpVersion":"HTTP/1.1","httpMethod":"GET","requestId":null}}

The expected extraction fields are not in the log, so the fallback option is unknown-aws-waf-<originating-host>. If the log originated from 8.8.8.8, then the hostname would be 8.8.8.8.

Imperva Incapsula

The hostname for Imperva Incapsula log events is determined by the name of the S3 bucket defined as the environment variable bucket_name in the /opt/armor/log-relay/conf.d/<pipeline_name>.<friendly_id>.env file on the Log Relay server.

If bucket_name was example-bucket-name, the hostname would be example-bucket-name. Be sure that only one Imperva Incapsula device feeds into one S3 bucket, or multiple devices will report under the same hostname.

  • No labels