Using the Datalake

What is the Datalake

The Armor data lake is a centralized repository for storing Armor collected data. With regards to CSPM, the data lake contains all the data for every report created for an environment and all the historical data from when the reports are run. This can be a lot of data so narrowing down the scope of information is critical to making sense of it all.

Accessing the Datalake

Users can access the datalake in two ways:

Option 1: Compliance in AMP

Select a Report from the Report List and click on it’s name to access the details page.
Then expand down to the control level of a section to view links for Remediation and Advanced Query.
Click on Advanced Query.
This opens ChaosSearch in a new window.
Click on the Single Sign On button.
Click Next again on the next page to sign in to ChaosSearch.
Once the page loads the following will show:
Note that there are two filters already being applied based on which control was open when Advanced Query was selected. The ruleId and ReportId.
To see the complete report, click on the X next to the rule.Id and now the filter is only using the ReportId to get data.
1. Keeping the rule.Id can also be useful for comparing changes over time (using a wider date range) for that rule.
Changing the date range allows for viewing a single or multiple runs of the report depending on the goal.

Option 2: Log Search in AMP

Select a Report from the Report List and click the report name to access the details.
Copy its unique report Id by navigating into the report’s detail page.
Navigate to Security -> Log Search and SSO into Chaos Search.
Create a filter by doing the following:
1. Click on Add filter.
2. In Field select event.ReportId
3. Select is for Operator.
4. Paste the report Id from the report details page into the Value field.
5. Click Save.
Now set the date range to encompass the report date or dates to show and click Refresh.

Data Presentation

Data consists of documents stored in the datalake. Each document contains all the data related to that particular rule and resource. Below are examples of the table and JSON views:

Table Example

Fields	Values
@timestamp	Nov 2, 2020 @ 17:27:23.779
@version	1
_id	5.83E+08
_index	1_4803_customer
_score	1
_type	doc
armor_metrics.input_port	5443
armor_metrics.latency.processing	0.112
armor_metrics.processing_chain	["KVN_V4_collector_i-095a2e7cd62db995c\|2020-11-02T23:27:23Z","KVN_V4_processor_i-09425dd816b437aeb\|2020-11-02T23:27:23Z"]
cloud.account.id	7.41E+11
cloud.instance.id	memcache-test-ind
cloud.machine.type	MEMCACHED
cloud.provider	aws
cloud.region	us-west-2
data_type	cspm-detections
document_size	1,819
event.ReportId	bafee260-1d44-11eb-a15a-eff990dadedf
event.ReportTitle	PCI DSS FOR R&D
event.ReportType	MANDATE
event.outcome	FAIL
event.reason	[Cluster ID, memcache-test-ind],[Subnet Group, default],[Vpc Id, vpc-95234ef0]
event.reference	https://portal.secure-stage.services/compliance/reports/controls/remediations/147
event.severity	2
event_uuid	b6611368-6641-4fcb-8b34-a999b3b07328
external_id	00000000-0000-0000-0000-000000004803
index_type	cspm-detections
labels.parent_id	1
logsource.origin	unknown
message_size	0
riginal_timestamp	Nov 2, 2020 @ 14:06:58.000
received_timestamp	Nov 2, 2020 @ 17:27:23.779
rule.Ctrl_Obj	Ensure that AWS ElastiCache Memcached clusters are not associated with default VPC
rule.Policy	Payment Card Industry Data Security Standard (PCI-DSS)
rule.Reqt_Lvl1	Regularly test security systems and processes
rule.Reqt_Lvl2	Use intrusion-detection and/or intrusion-prevention techniques to detect and/or prevent intrusions into the network.
rule.Reqt_Lvl3	Boundary Protection
rule.Reqt_Lvl4	Flaw Remediation
rule.Section_Lvl1	Requirement 11
rule.Section_Lvl2	11.4
rule.Section_Lvl3	SC-7
rule.Section_Lvl4	SI-2
rule.id	147
tags	["core_metadata_miss","customer","mismatched_tenant_external_id","cached_parent_metadata"]
tenant_id	4803
type	cspm

JSON Example

{
  "_score": 1,
  "_type": "doc",
  "_source": {
    "document_size": 1649,
    "event.reference": "https://portal.------.services/compliance/reports/controls/remediations/41",
    "rule.id": "41",
    "@timestamp": "2020-10-26T23:53:05.787Z",
    "tenant_id": "2177",
    "message_size": 0,
    "rule.Reqt_Lvl1": "Inventory of Authorized and Unauthorized Software",
    "cloud.instance.id": "sg-****",
    "_id": 30246894,
    "tags": "[\"cached_metadata_miss\",\"core_metadata_miss\",\"customer\",\"mismatched_tenant_external_id\",\"cached_parent_metadata\"]",
    "event.outcome": "PASS",
    "armor_metrics.processing_chain": "[\"KVN_V4_collector_i-095a2e7cd62db995c|2020-10-26T23:53:05Z\",\"KVN_V4_processor_i-0b1acc60b4ae2044b|2020-10-26T23:53:05Z\"]",
    "rule.Ctrl_Obj": "Ensure no security groups allow ingress from 0.0.0.0/0 to port 22",
    "armor_metrics.input_port": 5443,
    "original_timestamp": "2020-10-26T23:36:02.000Z",
    "logsource.origin": "unknown",
    "rule.Policy": "CIS Critical Security Controls (Top 20)",
    "rule.Reqt_Lvl2": "Continuous Monitoring",
    "cloud.machine.type": "VPC_SECURITY_GROUP",
    "rule.Section_Lvl2": "CA-7",
    "received_timestamp": "2020-10-26T23:53:05.787Z",
    "rule.Section_Lvl1": "CSC #2",
    "cloud.account.id": "********",
    "data_type": "cspm-detections",
    "event_uuid": "8dcccbb8-46d5-48e9-809f-5444d5579cc8",
    "event.severity": "8",
    "labels.parent_id": "1",
    "external_id": "00000000-0000-0000-0000-000000002177",
    "armor_metrics.latency.processing": 0.11045408248901367,
    "event.ReportId": "c1b36f40-125e-11eb-9963-b3d352dc1ad9",
    "event.ReportTitle": "CIS-TOP20",
    "type": "cspm",
    "armor_metadata.customer.cache_time": "2020-10-26T20:22:51.733Z",
    "@version": 1,
    "cloud.region": "us-east-1",
    "armor_metadata.customer.cache_expire": "1603916571.7332091",
    "cloud.provider": "aws",
    "event.ReportType": "MANDATE",
    "event.reason": "[VPC Id, vpc-*****]",
    "index_type": "cspm-detections"
  },
  "_id": "30246894",
  "_index": "1_2177_customer"
}

The schema for this document is based on Elastic Common Schema, please refer to the below links for the details and explanation of the fields:

cloud schema - https://www.elastic.co/guide/en/ecs/current/ecs-cloud.html
rule schema - https://www.elastic.co/guide/en/ecs/current/ecs-rule.html
- Custom Fields
  - Reqt_Lvl1 - Top level requirement
  - Section_Lvl1 - Top level section name
  - Reqt_Lvl2 - Second level requirement
  - Section_Lvl2 - Second level section name
  - Reqt_Lvl3 - Third level requirement
  - Section_Lvl3 - Third level section name
  - Reqt_Lvl4 - Fourth level requirement
  - Section_Lvl4 - Fourth level section name
  - Policy - the mandate selected at the time of report creation
event schema - https://www.elastic.co/guide/en/ecs/current/ecs-rule.html
- Custom Fields
  - Report Id - the unique Guid of the report generated
  - Report Title - the title used when creating the report

Helpful Fields for Searching the DataLake

Field	Filter By
cloud.provider	the cloud provider type (AWS, Google or Azure)
cloud.account.id	a specific cloud account Id as reports may contain more than one account
cloud.instance.id	the instance id
event.ReportId	a specific report id as multiple reports may exist
event.outcome	whether the resource Passed/Failed
rule.Policy	a specific policy

Addring a Filter

To add additional filters, click on the Add Filter Button.

Then set the field to one of the helpful fields above, select the operator, put in the value and hit save. The data is now filtered on a specific reportId, rPolicy or other field selected.

Viewing Datalake Aggregations

Please refer to https://armor-jira.atlassian.net/wiki/spaces/AKB/pages/907968868/Cloud+Security+Posture+Management#Dashboarding-and-Reporting for custom aggregations, visualizations and custom reports.