Using the Datalake

Searching the Datalake

The Datalake contains all the data for every report created for an environment and all the historical data from when the reports are run. This can be a lot of data so narrowing down the information for review is critical to making sense of the information.

The first thing to start with for viewing report data is to narrow down the search by the report needed. There are two ways to access the Datalake and the first is from within the CSPM page itself.

1. Select a Report from the Report List and click on it’s name to access the details page.

2. Then expand down to the control level of a section to view links for Remediation and Advanced Query.

3. Click on Advanced Query.

4. This opens ChaosSearch in a new window.

5. Click on the Single Sign On button.

6. Click Next again on the next page to sign in to ChaosSearch.

7. Once the page loads the following will show:

8. Note that there are two filters already being applied based on which control was open when Advanced Query was selected. The ruleId and ReportId.

9. To see the complete report, click on the X next to the rule.Id and now the filter is only using the ReportId to get data.

   1. Keeping the rule.Id can also be useful for comparing changes over time (using a wider date range) for that rule.

10. Changing the date range allows for viewing a single or multiple runs of the report depending on the goal.

The second way to access ChaosSearch is via the Log Source page in AMP.

1. Select a Report from the Report List and click on it’s name to access the details.

2. Copy its unique report Id by navigating into the report’s detail page.

3. Navigate to Security -> Log Search and SSO into Chaos Search.

Create a filter by doing the following:

1. Click on Add filter.

2. In Field select event.ReportId

3. Select is for Operator.

4. Paste the report Id from the report details page into the Value field.

5. Click Save.

Now set the date range to encompass the report date or dates to show and click Refresh.

{
  "_score": 1,
  "_type": "doc",
  "_source": {
    "document_size": 1649,
    "event.reference": "https://portal.------.services/compliance/reports/controls/remediations/41",
    "rule.id": "41",
    "@timestamp": "2020-10-26T23:53:05.787Z",
    "tenant_id": "2177",
    "message_size": 0,
    "rule.Reqt_Lvl1": "Inventory of Authorized and Unauthorized Software",
    "cloud.instance.id": "sg-****",
    "_id": 30246894,
    "tags": "[\"cached_metadata_miss\",\"core_metadata_miss\",\"customer\",\"mismatched_tenant_external_id\",\"cached_parent_metadata\"]",
    "event.outcome": "PASS",
    "armor_metrics.processing_chain": "[\"KVN_V4_collector_i-095a2e7cd62db995c|2020-10-26T23:53:05Z\",\"KVN_V4_processor_i-0b1acc60b4ae2044b|2020-10-26T23:53:05Z\"]",
    "rule.Ctrl_Obj": "Ensure no security groups allow ingress from 0.0.0.0/0 to port 22",
    "armor_metrics.input_port": 5443,
    "original_timestamp": "2020-10-26T23:36:02.000Z",
    "logsource.origin": "unknown",
    "rule.Policy": "CIS Critical Security Controls (Top 20)",
    "rule.Reqt_Lvl2": "Continuous Monitoring",
    "cloud.machine.type": "VPC_SECURITY_GROUP",
    "rule.Section_Lvl2": "CA-7",
    "received_timestamp": "2020-10-26T23:53:05.787Z",
    "rule.Section_Lvl1": "CSC #2",
    "cloud.account.id": "********",
    "data_type": "cspm-detections",
    "event_uuid": "8dcccbb8-46d5-48e9-809f-5444d5579cc8",
    "event.severity": "8",
    "labels.parent_id": "1",
    "external_id": "00000000-0000-0000-0000-000000002177",
    "armor_metrics.latency.processing": 0.11045408248901367,
    "event.ReportId": "c1b36f40-125e-11eb-9963-b3d352dc1ad9",
    "event.ReportTitle": "CIS-TOP20",
    "type": "cspm",
    "armor_metadata.customer.cache_time": "2020-10-26T20:22:51.733Z",
    "@version": 1,
    "cloud.region": "us-east-1",
    "armor_metadata.customer.cache_expire": "1603916571.7332091",
    "cloud.provider": "aws",
    "event.ReportType": "MANDATE",
    "event.reason": "[VPC Id, vpc-*****]",
    "index_type": "cspm-detections"
  },
  "_id": "30246894",
  "_index": "1_2177_customer"
}

The schema for this document is based on Elastic Common Schema, please refer to the below links for the details and explanation of the fields:

• cloud schema - https://www.elastic.co/guide/en/ecs/current/ecs-cloud.html

• rule schema - https://www.elastic.co/guide/en/ecs/current/ecs-rule.html

  • Custom Fields

    • Reqt_Lvl1 - Top level requirement

    • Section_Lvl1 - Top level section name

    • Reqt_Lvl2 - Second level requirement

    • Section_Lvl2 - Second level section name

    • Reqt_Lvl3 - Third level requirement

    • Section_Lvl3 - Third level section name

    • Reqt_Lvl4 - Fourth level requirement

    • Section_Lvl4 - Fourth level section name

    • Policy - the mandate selected at the time of report creation

• event schema - https://www.elastic.co/guide/en/ecs/current/ecs-rule.html

  • Custom Fields

    • Report Id - the unique Guid of the report generated

    • Report Title - the title used when creating the report

To add additional filters, click on the Add Filter Button.

Other helpful fields to search the datalake

• cloud.provider

• cloud.account.id

• cloud.instance.id

• event.ReportId

• rule.Policy

Viewing Datalake Aggregations

Please refer to https://armor-jira.atlassian.net/wiki/spaces/AKB/pages/907968868/Cloud+Security+Posture+Management#Dashboarding-and-Reporting for custom aggregations, visualizations and custom reports.