What is the Datalake
The Armor data lake is a centralized repository for storing Armor collected data. With regards to CSPM, the data lake contains all the data for every report created for an environment and all the historical data from when the reports are run. This can be a lot of data so narrowing down the scope of information is critical to making sense of it all.
Accessing the Datalake
Users can access the datalake in two ways:
| Expand | ||
|---|---|---|
| ||
|
...
| title | Option 2: Log Search in AMP |
|---|
- Select a Report from the Report List and click the report name to access the details.
- Copy its unique report Id by navigating into the report’s detail page.
- Navigate to Security -> Log Search and SSO into Chaos Search.
- Create a filter by doing the following:
- Click on Add filter.
- In Field select event.ReportId
- Select is for Operator.
- Paste the report Id from the report details page into the Value field.
- Click Save.
- Now set the date range to encompass the report date or dates to show and click Refresh.
Data Presentation
Data consists of documents stored in the datalake. Each document contains all the data related to that particular rule and resource. Below are examples of the table and JSON views:
...
| title | Table Example |
|---|
...
Fields
...
Values
...
@timestamp
...
Nov 2, 2020 @ 17:27:23.779
...
@version
...
1
...
_id
...
5.83E+08
...
_index
...
1_4803_customer
...
_score
...
1
...
_type
...
doc
...
armor_metrics.input_port
...
5443
...
armor_metrics.latency.processing
...
0.112
...
armor_metrics.processing_chain
...
["KVN_V4_collector_i-095a2e7cd62db995c|2020-11-02T23:27:23Z","KVN_V4_processor_i-09425dd816b437aeb|2020-11-02T23:27:23Z"]
...
...
7.41E+11
...
...
memcache-test-ind
...
cloud.machine.type
...
MEMCACHED
...
cloud.provider
...
aws
...
cloud.region
...
us-west-2
...
data_type
...
cspm-detections
...
document_size
...
1,819
...
event.ReportId
...
bafee260-1d44-11eb-a15a-eff990dadedf
...
event.ReportTitle
...
PCI DSS FOR R&D
...
event.ReportType
...
MANDATE
...
event.outcome
...
FAIL
...
event.reason
...
[Cluster ID, memcache-test-ind],[Subnet Group, default],[Vpc Id, vpc-95234ef0]
...
event.reference
...
https://portal.secure-stage.services/compliance/reports/controls/remediations/147
...
event.severity
...
2
...
event_uuid
...
b6611368-6641-4fcb-8b34-a999b3b07328
...
external_id
...
00000000-0000-0000-0000-000000004803
...
index_type
...
cspm-detections
...
labels.parent_id
...
1
...
logsource.origin
...
unknown
...
message_size
...
0
...
riginal_timestamp
...
Nov 2, 2020 @ 14:06:58.000
...
received_timestamp
...
Nov 2, 2020 @ 17:27:23.779
...
rule.Ctrl_Obj
...
Ensure that AWS ElastiCache Memcached clusters are not associated with default VPC
...
rule.Policy
...
Payment Card Industry Data Security Standard (PCI-DSS)
...
rule.Reqt_Lvl1
...
Regularly test security systems and processes
...
rule.Reqt_Lvl2
...
Use intrusion-detection and/or intrusion-prevention techniques to detect and/or prevent intrusions into the network.
...
rule.Reqt_Lvl3
...
Boundary Protection
...
rule.Reqt_Lvl4
...
Flaw Remediation
...
rule.Section_Lvl1
...
Requirement 11
...
rule.Section_Lvl2
...
11.4
...
rule.Section_Lvl3
...
SC-7
...
rule.Section_Lvl4
...
SI-2
...
...
147
...
tags
...
["core_metadata_miss","customer","mismatched_tenant_external_id","cached_parent_metadata"]
...
tenant_id
...
4803
...
type
...
cspm
| Expand | |||||||||
|---|---|---|---|---|---|---|---|---|---|
| |||||||||
|
The schema for this document is based on Elastic Common Schema, please refer to the below links for the details and explanation of the fields:
cloud schema - https://www.elastic.co/guide/en/ecs/current/ecs-cloud.html
rule schema - https://www.elastic.co/guide/en/ecs/current/ecs-rule.html
Custom Fields
Reqt_Lvl1 - Top level requirement
Section_Lvl1 - Top level section name
Reqt_Lvl2 - Second level requirement
Section_Lvl2 - Second level section name
Reqt_Lvl3 - Third level requirement
Section_Lvl3 - Third level section name
Reqt_Lvl4 - Fourth level requirement
Section_Lvl4 - Fourth level section name
Policy - the mandate selected at the time of report creation
event schema - https://www.elastic.co/guide/en/ecs/current/ecs-rule.html
Custom Fields
Report Id - the unique Guid of the report generated
Report Title - the title used when creating the report
Helpful Fields for Searching the DataLake
...
Field
...
Filter By
...
cloud.provider
...
the cloud provider type (AWS, Google or Azure)
...
...
a specific cloud account Id as reports may contain more than one account
...
...
the instance id
...
event.ReportId
...
a specific report id as multiple reports may exist
...
event.outcome
...
whether the resource Passed/Failed
...
rule.Policy
...
a specific policy
Addring a Filter
To add additional filters, click on the Add Filter Button.
Then set the field to one of the helpful fields above, select the operator, put in the value and hit save. The data is now filtered on a specific reportId, rPolicy or other field selected.
Viewing Datalake Aggregations
Please refer to https://armor-jira.atlassian.net/wiki/spaces/AKB/pages/907968868/Cloud+Security+Posture+Management#Dashboarding-and-Reporting for custom aggregations, visualizations and custom reports.