Azure Event Hubs

Event Hubs is a fully managed, real-time data ingestion service that’s simple, trusted, and scalable. Stream millions of events per second from any source to build dynamic data pipelines and immediately respond to business challenges. Keep processing data during emergencies using the geo-disaster recovery and geo-replication features.

Integrate seamlessly with other Azure services to unlock valuable insights. Allow existing Apache Kafka clients and applications to talk to Event Hubs without any code changes—you get a managed Kafka experience without having to manage your own clusters. Experience real-time data ingestion and microbatching on the same stream.

Open

How It Works

You can use this document to learn how to create Microsoft Azure Event Hubs as an External Log Source. 

Armor supports log collection for Microsoft Azure services listed below:

• Azure Event Hubs

• Azure Application Gateway

In this article:

Prerequisites:

Event Hubs and Storage Account Creation

Azure Event Hubs is a big data streaming platform and event ingestion service. It can receive and process millions of events per second.

Subscription

Create Subscription (if not available) - This should be controlled by the billing administrator of the Azure Directory.

Resource Group

Create Resource Group (if not available) - A resource group is a container that holds related resources for an Azure solution. The resource group can include all the resources for the solution, or only those resources that you want to manage as a group.

Create an Event Hubs Namespace

1. Log in to the Azure Portal. (https://portal.azure.com/#allservices)

2. In search box next to All services, type event hub when Event Hub appear in the search results, select it.

3. Event Hubs -> +Add
   

4. On the Create namespace page, take the following steps:

5. Select the subscription in which you want to create the namespace.

6. Select the resource group you created in the previous step.

7. Enter a name for the namespace. The system immediately checks to see if the name is available.

8. Select a location for the namespace.

9. Choose the pricing tier(Standard (can not be basic as we are required to provide a named ConsumerGroup)).

10. Throughput Units: 1

    1. Throughput units are explicitly selected by the customer, either through the Azure portal or event hub management APIs. Throughput units apply to all event hubs in a namespace, and each throughput unit entitles the namespace to the following capabilities:

    2. Up to 1 MB per second of ingress events (= events send into an event hub), but no more than 1,000 ingress events, management operations, or control API calls per second.

    3. Up to 2 MB per second of egress events (= events consumed from an event hub).

    4. Up to 84 GB of event storage (sufficient for the default 24-hour retention period).

    5. Throughput units are billed hourly, based on the maximum number of units selected during this hour.

11. Setup Features

    1. Provides enhanced availability by spreading replicas across availability zones within one region at no additional cost. Learn more
       
       

       Open

       

12. Add tags: Tags can be any additional metadata used to describe the resource

13. Review and create
    
    
    

Create an Event Hub

1. On the Event Hubs Namespace page, select Event Hubs in the left menu.

2. At the top of the window, click + Event Hub.
   
   

3. Type a name - armor-logs for your event hub, then click Create.

4. Click Create
   
   
   

Create Shared access policies

1. On the Event Hubs Namespace page, select Event Hubs in the left menu.

2. Select Event Hub created above, armor-logs.

3. Select Shared access policies in Settings in the left menu and click +Add

   1. Enter Policy name → armor-logs

   2. Select Manage

   3. Click Create
      
      

Create a Consumer Group

1. On the Event Hubs Namespace page, Click Event Hubs in the left menu.

2. Select Event Hub created above, armor-logs.

3. Select Consumer groups in Entities and click +Consumer group

   1. Name → armor-logs

   2. Click Create
      
      
      

Retrieve Event Hub Connection String

1. Select All services, then type event hub when Event hub appear in the search results, select it.

2. On the Event Hubs Namespace page, select Event Hubs.

3. Select Event Hub created above, armor-logs.

4. Select Shared access policies in settings.

5. Select the Shared access policy created above, armor-logs.

6. Copy Connection string—primary key
   
   
   

Create Storage Account

1. Log in to the Azure Portal. (https://portal.azure.com/#allservices )

2. In search box next to All services, type storage account when Storage accounts appear in the search results, select it. All Services ->Storage accounts -> +Add
   

3. Basic :

   1. Storage account name

   2. Performance: Depending on the type of storage account you create, you can choose between standard and premium performance tiers.Learn more

   3. Account kind: Azure Storage offers several types of storage accounts. Each type supports different features and has its own pricing model.Learn more

4. Networking:

   1. Public endpoint (all networks)
      

5. Advanced:

   1. Secure transfer required: Enabled

   2. Large file shares: Disabled

   3. Blob soft delete: Disabled

   4. Versioning: Disabled

   5. Hierarchical namespace: Disabled
      

6. Tags: Tags are name/value pairs that enable you to categorize resources and view consolidated billing by applying the same tag to multiple resources and resource groups
   
   
   

7. Review and Create

8. Click Create

Retrieve Storage Account Connection String

1. Select All services, then type storage account when Storage account appear in the search results, select it. Click on the storage account → Settings → Access Keys

2. From the key 1 or key 2 section copy the Connection string
   
   
   

Create Armor Azure Event Hubs Log Source

1. In the Armor Management Portal (AMP), in the left-side navigation, click Security.

2. Click Log & Data Management.

3. Click External Sources.

4. Click the plus ( + ) sign.

   • If you do not have any log sources already created, then click Add a New Log Source.

5. Complete the missing fields:

   • In Endpoint, select the available Armor Endpoint.

   • In Log Source Type, select the Microsoft Azure Platform.

   • In HostName, enter a hostname

6. In Protocol, confirm that the Azure Platform is selected.

7. Enter your Azure platform details:

   • Consumer Group, by default, armor-logs will be populated.

   • In Event Hub Connection String, paste your event hub connection string

   • In Storage Acc. Connection String, paste your storage account connection string

8. Click Save Log Source.

9. A message will display at the bottom of the screen, indicating that the log source has been created.