/
Azure Event Hubs

Azure Event Hubs

 

Event Hubs is a fully managed, real-time data ingestion service that’s simple, trusted, and scalable. Stream millions of events per second from any source to build dynamic data pipelines and immediately respond to business challenges. Keep processing data during emergencies using the geo-disaster recovery and geo-replication features.

Integrate seamlessly with other Azure services to unlock valuable insights. Allow existing Apache Kafka clients and applications to talk to Event Hubs without any code changes—you get a managed Kafka experience without having to manage your own clusters. Experience real-time data ingestion and microbatching on the same stream.

How It Works

 

 

Features

Ingest millions of events per second

Continuously ingress data from hundreds of thousands of sources with low latency and configurable time retention.

Enable real-time and micro-batch processing concurrently

Seamlessly send data to Blob storage or Data Lake Storage for long-term retention or micro-batch processing with Event Hubs Capture.

Get a managed service with elastic scale

Easily scale from streaming megabytes of data to terabytes while keeping control over when and how much to scale.

Easily connect with the Apache Kafka ecosystem

Seamlessly connect Event Hubs with your Kafka applications and clients with Azure Event Hubs for Apache Kafka®.

Build a serverless streaming solution

Natively connect with Stream Analytics to build an end-to-end serverless streaming solution.

Ingest events on Azure Stack Hub and realize hybrid cloud solutions

Locally ingest and process data at a large scale on your Azure Stack Hub and implement hybrid cloud architectures by leveraging Azure services to further process, visualize, or store your data.

 

 

You can use this document to learn how to create Microsoft Azure Event Hubs as an External Log Source. 

Armor supports log collection for Microsoft Azure services listed below:

 

In this article:

Prerequisites:

Permissions

Your account must have the following permissions:

  • Read Log Management

  • Write Log Management

  • Delete Log Management

To learn more about permissions, see Roles and Permissions.

Microsoft Azure Portal:

  1. An Azure account with an active subscription

  2. Azure Event Hubs connection string and a Azure Storage Account connection string. To learn more about how to create and configure EventHubs and Storage account, see the article here.

Event Hubs and Storage Account Creation

Azure Event Hubs is a big data streaming platform and event ingestion service. It can receive and process millions of events per second.

Subscription

Create Subscription (if not available) - This should be controlled by the billing administrator of the Azure Directory.

Resource Group

Create Resource Group (if not available) - A resource group is a container that holds related resources for an Azure solution. The resource group can include all the resources for the solution, or only those resources that you want to manage as a group.

Create an Event Hubs Namespace

The following steps take place inside the Azure portal.

  1. Log in to the Azure Portal. (https://portal.azure.com/#allservices)

  2. In search box next to All services, type event hub when Event Hub appear in the search results, select it.

  3. Event Hubs -> +Add

  4. On the Create namespace page, take the following steps:

  5. Select the subscription in which you want to create the namespace.

  6. Select the resource group you created in the previous step.

  7. Enter a name for the namespace. The system immediately checks to see if the name is available.

  8. Select a location for the namespace.

  9. Choose the pricing tier(Standard (can not be basic as we are required to provide a named ConsumerGroup)).

  10. Throughput Units: 1

    1. Throughput units are explicitly selected by the customer, either through the Azure portal or event hub management APIs. Throughput units apply to all event hubs in a namespace, and each throughput unit entitles the namespace to the following capabilities:

    2. Up to 1 MB per second of ingress events (= events send into an event hub), but no more than 1,000 ingress events, management operations, or control API calls per second.

    3. Up to 2 MB per second of egress events (= events consumed from an event hub).

    4. Up to 84 GB of event storage (sufficient for the default 24-hour retention period).

    5. Throughput units are billed hourly, based on the maximum number of units selected during this hour.

  11. Setup Features

    1. Provides enhanced availability by spreading replicas across availability zones within one region at no additional cost. Learn more

  12. Add tags: Tags can be any additional metadata used to describe the resource

  13. Review and create



Create an Event Hub

  1. On the Event Hubs Namespace page, select Event Hubs in the left menu.

  2. At the top of the window, click + Event Hub.

  3. Type a name - armor-logs for your event hub, then click Create.

  4. Click Create