Using the Datalake for Vulnerabilities

Owned by Aaron Gabriel (Deactivated)
Last updated: 15 December 2023 by Aaron Gabriel (Deactivated)
6 min read

 

Understanding The Datalake

The Armor data lake is a centralized repository for storing Armor collected data. With regards to vulnerabilities, the data lake contains all the data for every report created for an environment and all the historical data from when the reports are run. This can be a lot of data so narrowing down the scope of information is critical to making sense of it all.

 

Accessing the Datalake

Users can access the datalake in two ways:

  1. Select a Report from the Report List and click on it’s name to access the details page.

  2. Then expand down to the control level of a section to view links for Remediation and Advanced Query.

  3. Click on Advanced Query.

  4. This opens ChaosSearch in a new window.

  5. Click on the Single Sign On button.

  6. Click Next again on the next page to sign in to ChaosSearch.

  7. Once the page loads the following will show:

  8. Note that there are two filters already being applied based on which control was open when Advanced Query was selected. The ruleId and ReportId.

  9. To see the complete report, click on the X next to the rule.Id and now the filter is only using the ReportId to get data.

    1. Keeping the rule.Id can also be useful for comparing changes over time (using a wider date range) for that rule.

  10. Changing the date range allows for viewing a single or multiple runs of the report depending on the goal.

  1. Select a Report from the Report List and click the report name to access the details. 

  2. Copy its unique report Id by navigating into the report’s detail page.

  3. Navigate to Security -> Log Search and SSO into Chaos Search.

  1. Create a filter by doing the following:

    1. Click on Add filter.

    2. In Field select event.ReportId

    3. Select is for Operator.

    4. Paste the report Id from the report details page into the Value field.

    5. Click Save.

  2. Now set the date range to encompass the report date or dates to show and click Refresh.

 

Data Presentation

Data consists of documents stored in the datalake. Each document contains all the data related to that particular rule and resource. Below are examples of the table and JSON views:

FIELD

VALUES

FIELD

VALUES

1

@timestamp

Nov 25, 2020 @ 07:32:27.480

2

#@version

1

3

_id

47741608

4

_index

1_1024_customer

5

#_score

1

6

_type

doc

7

armor_metrics.input_port

5445

8

armor_metrics.latency.processing

0.857

9

armor_metrics.processing_chain

["KVN_V4_collector_i-0908b8b2b53868dc0|2020-11-25T13:32:27Z","KVN_V4_processor_i-0aa172c88f440b715|2020-11-25T13:32:28Z"]

10

document_size

3,926

11

event_uuid

6d820110-73e5-45c9-945e-10c281fd4cb4

12

external_id

4f5b9ab7-8e57-4993-b0fb-440cd44d11e5

13

host.hostname

ip-10-0-0-8.us-west-2.compute.internal

14

host.ip

10.0.0.8

15

host.os.full

Amazon Linux 2

16

host.os.name

Linux

17

host_asset_id

75424166

18

index_type

ecs-1.5.0-vulnerability

19

labels.parent_id

1

20

logsource.origin

unknown

21

message_size

0

22

original_timestamp

Nov 25, 2020 @ 07:32:27.233

23

received_timestamp

Nov 25, 2020 @ 07:32:27.480

24

tags

["customer","flow_source_data_miss","default_parent_id","cached_parent_metadata"]

25

tenant_id

1024

26

vulnerability.category

["AMAZON LINUX","PCI"]

27

vulnerability.consequence

Allows unauthorized disclosure of information; allows unauthorized modification; allows disruption of service.

28

vulnerability.cve

http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-5188

29

vulnerability.description

Amazon Linux Security Advisory for e2fsprogs: ALAS2-2020-1509

30

vulnerability.diagnosis

&lt;DIV&gt; Issue Overview: <P>An exploitable code execution vulnerability exists in the quota file functionality of E2fsprogs 1.45.3. A specially crafted ext4 partition can cause an out-of-bounds write on the heap, resulting in code execution. An attacker can corrupt a partition to trigger this vulnerability. (<A HREF="https://access.redhat.com/security/cve/CVE-2019-5094" TARGET="_blank">CVE-2019-5094 </A>)</P><P>A code execution vulnerability exists in the directory rehashing functionality of E2fsprogs e2fsck 1.45.4. A specially crafted ext4 directory can cause an out-of-bounds write on the stack, resulting in code execution. An attacker can corrupt a partition to trigger this vulnerability. (<A HREF="https://access.redhat.com/security/cve/CVE-2019-5188" TARGET="_blank">CVE-2019-5188 </A>)</P> &lt;/DIV&gt;

31

vulnerability.discovery

0

32

vulnerability.enumeration

135

33

vulnerability.id

352127

34

vulnerability.last_modification

Oct 29, 2020 @ 07:29:25.000

35

vulnerability.patchable

1

36

vulnerability.pci_flag

1

37

vulnerability.published

Oct 29, 2020 @ 07:29:25.000

38

vulnerability.reference

https://alas.aws.amazon.com/AL2/ALAS-2020-1509.html

39

vulnerability.report_id

20201125.133227

40

vulnerability.results

Package Installed Version Required Version e2fsprogs 1.42.9-12.amzn2.0.2.x86_64 1.42.9-19.amzn2 e2fsprogs-libs 1.42.9-12.amzn2.0.2.x86_64 1.42.9-19.amzn2 libcom_err 1.42.9-12.amzn2.0.2.x86_64 1.42.9-19.amzn2 libss 1.42.9-12.amzn2.0.2.x86_64 1.42.9-19.amzn2 e2fsprogs 1.42.9-12.amzn2.0.2.x86_64 1.42.9-19.amzn2

41

vulnerability.scanner.vendor

Powered by Qualys

42

vulnerability.score.base

6.7

43

vulnerability.score.environmental

0.0

44

vulnerability.score.temporal

5.4

45

vulnerability.score.version

3.0

46

vulnerability.severity

3

47

vulnerability.solution

Please refer to Amazon advisory <A HREF="https://alas.aws.amazon.com/AL2/ALAS-2020-1509.html" TARGET="_blank">ALAS-2020-1509</A> for affected packages and patching details, or update with your package manager. <P>Patch:<BR> Following are links for downloading patches to fix the vulnerabilities: <P> <A HREF="https://alas.aws.amazon.com/AL2/ALAS-2020-1509.html" TARGET="_blank">ALAS-2020-1509: Amazon Linux 2 (e2fsprogs (1.42.9-19.amzn2) on x86_64)</A><P> <A HREF="https://alas.aws.amazon.com/AL2/ALAS-2020-1509.html" TARGET="_blank">ALAS-2020-1509: Amazon Linux 2 (e2fsprogs (1.42.9-19.amzn2) on aarch64)</A><P> <A HREF="https://alas.aws.amazon.com/AL2/ALAS-2020-1509.html" TARGET="_blank">ALAS-2020-1509: Amazon Linux 2 (e2fsprogs (1.42.9-19.amzn2) on src)</A><P> <A HREF="https://alas.aws.amazon.com/AL2/ALAS-2020-1509.html" TARGET="_blank">ALAS-2020-1509: Amazon Linux 2 (e2fsprogs (1.42.9-19.amzn2) on i686)</A>

48

vulnerability.status

Active

49

vulnerability.vulnerability_type

VULNERABILITY

{ "_score": 1, "_type": "doc", "_source": { "vulnerability.enumeration": "135", "document_size": 3926, "@timestamp": "2020-11-25T13:32:27.480Z", "vulnerability.published": "2020-10-29T12:29:25.000Z", "vulnerability.results": "Package\tInstalled Version\tRequired Version\ne2fsprogs\t1.42.9-12.amzn2.0.2.x86_64\t1.42.9-19.amzn2\ne2fsprogs-libs\t1.42.9-12.amzn2.0.2.x86_64\t1.42.9-19.amzn2\nlibcom_err\t1.42.9-12.amzn2.0.2.x86_64\t1.42.9-19.amzn2\nlibss\t1.42.9-12.amzn2.0.2.x86_64\t1.42.9-19.amzn2\ne2fsprogs\t1.42.9-12.amzn2.0.2.x86_64\t1.42.9-19.amzn2", "tenant_id": "1024", "vulnerability.cve": "http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-5188", "host.os.name": "Linux", "message_size": 0, "vulnerability.description": "Amazon Linux Security Advisory for e2fsprogs: ALAS2-2020-1509", "vulnerability.scanner.vendor": "Powered by Qualys", "_id": 47741608, "tags": "[\"customer\",\"flow_source_data_miss\",\"default_parent_id\",\"cached_parent_metadata\"]", "armor_metrics.processing_chain": "[\"KVN_V4_collector_i-0908b8b2b53868dc0|2020-11-25T13:32:27Z\",\"KVN_V4_processor_i-0aa172c88f440b715|2020-11-25T13:32:28Z\"]", "vulnerability.score.temporal": "5.4", "vulnerability.solution": "Please refer to Amazon advisory <A HREF=\"https://alas.aws.amazon.com/AL2/ALAS-2020-1509.html\" TARGET=\"_blank\">ALAS-2020-1509</A> for affected packages and patching details, or update with your package manager.\n<P>Patch:<BR>\nFollowing are links for downloading patches to fix the vulnerabilities:\n<P> <A HREF=\"https://alas.aws.amazon.com/AL2/ALAS-2020-1509.html\" TARGET=\"_blank\">ALAS-2020-1509: Amazon Linux 2 (e2fsprogs (1.42.9-19.amzn2) on x86_64)</A><P> <A HREF=\"https://alas.aws.amazon.com/AL2/ALAS-2020-1509.html\" TARGET=\"_blank\">ALAS-2020-1509: Amazon Linux 2 (e2fsprogs (1.42.9-19.amzn2) on aarch64)</A><P> <A HREF=\"https://alas.aws.amazon.com/AL2/ALAS-2020-1509.html\" TARGET=\"_blank\">ALAS-2020-1509: Amazon Linux 2 (e2fsprogs (1.42.9-19.amzn2) on src)</A><P> <A HREF=\"https://alas.aws.amazon.com/AL2/ALAS-2020-1509.html\" TARGET=\"_blank\">ALAS-2020-1509: Amazon Linux 2 (e2fsprogs (1.42.9-19.amzn2) on i686)</A>", "armor_metrics.input_port": "5445", "original_timestamp": "2020-11-25T13:32:27.233Z", "logsource.origin": "unknown", "vulnerability.score.environmental": "0.0", "vulnerability.status": "Active", "vulnerability.category": "[\"AMAZON LINUX\",\"PCI\"]", "host.ip": "10.0.0.8", "vulnerability.discovery": "0", "vulnerability.reference": "https://alas.aws.amazon.com/AL2/ALAS-2020-1509.html", "vulnerability.report_id": "20201125.133227", "received_timestamp": "2020-11-25T13:32:27.480Z", "host.os.full": "Amazon Linux 2", "vulnerability.pci_flag": "1", "vulnerability.patchable": "1", "vulnerability.score.version": "3.0", "event_uuid": "6d820110-73e5-45c9-945e-10c281fd4cb4", "vulnerability.last_modification": "2020-10-29T12:29:25.000Z", "vulnerability.diagnosis": "<DIV>\n Issue Overview:\n <P>An exploitable code execution vulnerability exists in the quota file functionality of E2fsprogs 1.45.3. A specially crafted ext4 partition can cause an out-of-bounds write on the heap, resulting in code execution. An attacker can corrupt a partition to trigger this vulnerability. (<A HREF=\"https://access.redhat.com/security/cve/CVE-2019-5094\" TARGET=\"_blank\">CVE-2019-5094 </A>)</P><P>A code execution vulnerability exists in the directory rehashing functionality of E2fsprogs e2fsck 1.45.4. A specially crafted ext4 directory can cause an out-of-bounds write on the stack, resulting in code execution. An attacker can corrupt a partition to trigger this vulnerability. (<A HREF=\"https://access.redhat.com/security/cve/CVE-2019-5188\" TARGET=\"_blank\">CVE-2019-5188 </A>)</P>\n </DIV>\n\n ", "labels.parent_id": "1", "host_asset_id": "75424166", "vulnerability.vulnerability_type": "VULNERABILITY", "external_id": "4f5b9ab7-8e57-4993-b0fb-440cd44d11e5", "vulnerability.score.base": "6.7", "armor_metrics.latency.processing": 0.8566529750823975, "vulnerability.severity": "3", "vulnerability.consequence": "Allows unauthorized disclosure of information; allows unauthorized modification; allows disruption of service.", "@version": 1, "host.hostname": "ip-10-0-0-8.us-west-2.compute.internal", "index_type": "ecs-1.5.0-vulnerability", "vulnerability.id": "352127" }, "_id": "47741608", "_index": "1_1024_customer" }

The schema for these documents is based on Elastic Common Schema, please refer to the below links for the details and explanation of the fields:

Vulnerability schema - https://www.elastic.co/guide/en/ecs/1.5/ecs-vulnerability.html

Custom Fields:

  • vulnerability.published - the date an entry for the vulnerability was given a CVE

  • vulnerability.results - the criteria used to determine the presence of the vulnerability

  • vulnerability.cve - contains a link to the vulnerability's entry in the CVE database

  • vulnerability.solution - provides instructions, if any exist, for remediating the vulnerability

  • vulnerability.status - lists New if it is the first time a vulnerability is detected by a scan; Active if the vulnerability was been detected by two or more scans; Fixed if the vulnerability was detected in the previous scan but the most recent scan shows it as fixed; and Re-Opened if the vulnerability was verified fixed previously but is no longer so

  • vulnerability.first_found - the date of the first scan in which the vulnerability was detected for a given server

  • vulnerability.last_found - the date of the most recent scan in which the vulnerability was detected for a given server

  • vulnerability.discovery - indicates whether the vulnerability was discovered through remote and/or authenticated scanning

  • vulnerability.pci_flag - a flag that indicates whether the vulnerability must be fixed to pass PCI compliance

  • vulnerability.patchable - contains a 1 if the vulnerability can be patched and a 0 if no patches currently exist for it

  • vulnerability.last_modification - the date of the vulnerability attributes' (title, severity level, patch availability, CVSS scores, PCI relevance, etc.) last modification

  • vulnerability.diagnosis - gives information about the technical details of the vulnerability, affected packages, severity scoring, and detection

  • vulnerability.vulnerability_type - indicates whether the detection was a potential vulnerability (vulnerabilities that cannot be fully verified but have at least one necessary condition for the vulnerability) or a vulnerability (the vulnerability can be fully verified)

  • vulnerability.consequences - provides information about the access an attacker who successfully exploits the vulnerability might gain

 

Helpful Fields for Searching The Datalake

FIELD

FILTER BY

FIELD

FILTER BY

vulnerability.category

The type of system or architecture that the vulnerability affects. See https://qualysguard.qualys.com/qwebhelp/fo_portal/knowledgebase/vulnerability_categories.htm#P for a listing of potential categories

vulnerability.severity

1 through 5

host.hostname

the hostname of any servers in your account

vulnerability.report_id

a scan ID that can be used to show only the vulnerabilities associated with a specific scan

 

 

Adding a Filter

To add additional filters, click on the Add Filter Button.

Then set the field to one of the helpful fields above, select the operator, put in the value and hit save. The data is now filtered on a specific reportId, rPolicy or other field selected.

 

Viewing Datalake Aggregations

Please refer to Reports for custom aggregations, visualizations and custom reports.