You are viewing an old version of this page. View the current version.
Compare with Current
View Page History
« Previous
Version 4
Next »
Understanding the Datalake
The Armor data lake is a centralized repository for storing Armor collected data. With regards to vulnerabilities, the data lake contains all the data for every report created for an environment and all the historical data from when the reports are run. This can be a lot of data so narrowing down the scope of information is critical to making sense of it all.
Accessing the Datalake
Users can access the datalake in two ways:
Option 1: Compliance in AMP
Select a Report from the Report List and click on it’s name to access the details page.
Then expand down to the control level of a section to view links for Remediation and Advanced Query.
Click on Advanced Query.
This opens ChaosSearch in a new window.
Click on the Single Sign On button.
Click Next again on the next page to sign in to ChaosSearch.
Once the page loads the following will show:
Note that there are two filters already being applied based on which control was open when Advanced Query was selected. The ruleId and ReportId.
To see the complete report, click on the X next to the rule.Id and now the filter is only using the ReportId to get data.
Keeping the rule.Id can also be useful for comparing changes over time (using a wider date range) for that rule.
Changing the date range allows for viewing a single or multiple runs of the report depending on the goal.
Data Presentation
Data consists of documents stored in the datalake. Each document contains all the data related to that particular rule and resource. Below are examples of the table and JSON views:
Table Example
Fields | Values |
---|
@timestamp | Nov 2, 2020 @ 17:27:23.779 |
@version | 1 |
_id | 5.83E+08 |
_index | 1_4803_customer |
_score | 1 |
_type | doc |
armor_metrics.input_port | 5443 |
armor_metrics.latency.processing | 0.112 |
armor_metrics.processing_chain | ["KVN_V4_collector_i-095a2e7cd62db995c|2020-11-02T23:27:23Z","KVN_V4_processor_i-09425dd816b437aeb|2020-11-02T23:27:23Z"] |
cloud.account.id | 7.41E+11 |
cloud.instance.id | memcache-test-ind |
cloud.machine.type | MEMCACHED |
cloud.provider | aws |
cloud.region | us-west-2 |
data_type | cspm-detections |
document_size | 1,819 |
event.ReportId | bafee260-1d44-11eb-a15a-eff990dadedf |
event.ReportTitle | PCI DSS FOR R&D |
event.ReportType | MANDATE |
event.outcome | FAIL |
event.reason | [Cluster ID, memcache-test-ind],[Subnet Group, default],[Vpc Id, vpc-95234ef0] |
event.reference | https://portal.secure-stage.services/compliance/reports/controls/remediations/147 |
event.severity | 2 |
event_uuid | b6611368-6641-4fcb-8b34-a999b3b07328 |
external_id | 00000000-0000-0000-0000-000000004803 |
index_type | cspm-detections |
labels.parent_id | 1 |
logsource.origin | unknown |
message_size | 0 |
riginal_timestamp | Nov 2, 2020 @ 14:06:58.000 |
received_timestamp | Nov 2, 2020 @ 17:27:23.779 |
rule.Ctrl_Obj | Ensure that AWS ElastiCache Memcached clusters are not associated with default VPC |
rule.Policy | Payment Card Industry Data Security Standard (PCI-DSS) |
rule.Reqt_Lvl1 | Regularly test security systems and processes |
rule.Reqt_Lvl2 | Use intrusion-detection and/or intrusion-prevention techniques to detect and/or prevent intrusions into the network. |
rule.Reqt_Lvl3 | Boundary Protection |
rule.Reqt_Lvl4 | Flaw Remediation |
rule.Section_Lvl1 | Requirement 11 |
rule.Section_Lvl2 | 11.4 |
rule.Section_Lvl3 | SC-7 |
rule.Section_Lvl4 | SI-2 |
rule.id | 147 |
tags | ["core_metadata_miss","customer","mismatched_tenant_external_id","cached_parent_metadata"] |
tenant_id | 4803 |
type | cspm |
JSON Example
{
"_score": 1,
"_type": "doc",
"_source": {
"document_size": 1649,
"event.reference": "https://portal.------.services/compliance/reports/controls/remediations/41",
"rule.id": "41",
"@timestamp": "2020-10-26T23:53:05.787Z",
"tenant_id": "2177",
"message_size": 0,
"rule.Reqt_Lvl1": "Inventory of Authorized and Unauthorized Software",
"cloud.instance.id": "sg-****",
"_id": 30246894,
"tags": "[\"cached_metadata_miss\",\"core_metadata_miss\",\"customer\",\"mismatched_tenant_external_id\",\"cached_parent_metadata\"]",
"event.outcome": "PASS",
"armor_metrics.processing_chain": "[\"KVN_V4_collector_i-095a2e7cd62db995c|2020-10-26T23:53:05Z\",\"KVN_V4_processor_i-0b1acc60b4ae2044b|2020-10-26T23:53:05Z\"]",
"rule.Ctrl_Obj": "Ensure no security groups allow ingress from 0.0.0.0/0 to port 22",
"armor_metrics.input_port": 5443,
"original_timestamp": "2020-10-26T23:36:02.000Z",
"logsource.origin": "unknown",
"rule.Policy": "CIS Critical Security Controls (Top 20)",
"rule.Reqt_Lvl2": "Continuous Monitoring",
"cloud.machine.type": "VPC_SECURITY_GROUP",
"rule.Section_Lvl2": "CA-7",
"received_timestamp": "2020-10-26T23:53:05.787Z",
"rule.Section_Lvl1": "CSC #2",
"cloud.account.id": "********",
"data_type": "cspm-detections",
"event_uuid": "8dcccbb8-46d5-48e9-809f-5444d5579cc8",
"event.severity": "8",
"labels.parent_id": "1",
"external_id": "00000000-0000-0000-0000-000000002177",
"armor_metrics.latency.processing": 0.11045408248901367,
"event.ReportId": "c1b36f40-125e-11eb-9963-b3d352dc1ad9",
"event.ReportTitle": "CIS-TOP20",
"type": "cspm",
"armor_metadata.customer.cache_time": "2020-10-26T20:22:51.733Z",
"@version": 1,
"cloud.region": "us-east-1",
"armor_metadata.customer.cache_expire": "1603916571.7332091",
"cloud.provider": "aws",
"event.ReportType": "MANDATE",
"event.reason": "[VPC Id, vpc-*****]",
"index_type": "cspm-detections"
},
"_id": "30246894",
"_index": "1_2177_customer"
}
The schema for these documents is based on Elastic Common Schema, please refer to the below links for the details and explanation of the fields:
Vulnerability schema - https://www.elastic.co/guide/en/ecs/1.5/ecs-vulnerability.html
Custom Fields:
- vulnerability.published - the date an entry for the vulnerability was given a CVE
- vulnerability.results - the criteria used to determine the presence of the vulnerability
- vulnerability.cve - contains a link to the vulnerability's entry in the CVE database
- vulnerability.solution - provides instructions, if any exist, for remediating the vulnerability
- vulnerability.status - lists New if it is the first time a vulnerability is detected by a scan; Active if the vulnerability was been detected by two or more scans; Fixed if the vulnerability was detected in the previous scan but the most recent scan shows it as fixed; and Re-Opened if the vulnerability was verified fixed previously but is no longer so
- vulnerability.first_found - the date of the first scan in which the vulnerability was detected for a given server
- vulnerability.last_found - the date of the most recent scan in which the vulnerability was detected for a given server
- vulnerability.discovery - indicates whether the vulnerability was discovered through remote and/or authenticated scanning
- vulnerability.pci_flag - a flag that indicates whether the vulnerability must be fixed to pass PCI compliance
- vulnerability.patchable - contains a 1 if the vulnerability can be patched and a 0 if no patches currently exist for it
- vulnerability.last_modification - the date of the vulnerability attributes' (title, severity level, patch availability, CVSS scores, PCI relevance, etc.) last modification
- vulnerability.diagnosis - gives information about the technical details of the vulnerability, affected packages, severity scoring, and detection
- vulnerability.vulnerability_type - indicates whether the detection was a potential vulnerability (vulnerabilities that cannot be fully verified but have at least one necessary condition for the vulnerability) or a vulnerability (the vulnerability can be fully verified)
- vulnerability.consequences - provides information about the access an attacker who successfully exploits the vulnerability might gain
Helpful Fields for Searching the Datalake
Adding a Filter
To add additional filters, click on the Add Filter Button.
Then set the field to one of the helpful fields above, select the operator, put in the value and hit save. The data is now filtered on a specific reportId, rPolicy or other field selected.
Viewing Datalake Aggregations
Please refer to Reports for custom aggregations, visualizations and custom reports.