Azure Data Explorer

This article may rely excessively on sources too closely associated with the subject, potentially preventing the article from being verifiable and neutral. Please help improve it by replacing them with more appropriate citations to reliable, independent, third-party sources. (April 2020) (Learn how and when to remove this message)

Azure Data Explorer

Developer(s)	Microsoft
Initial release	2018; 6 years ago (2018)
Platform	Microsoft Azure
Type	Cloud storage
License	Proprietary
Website	docs.microsoft.com/en-us/azure/data-explorer/

Azure Data Explorer is a fully-managed^[1] big data analytics cloud platform^[2][3] and data-exploration service,^[4] developed by Microsoft,^[5][6] that ingests structured, semi-structured (like JSON) and unstructured data (like free-text).^{[7][8][9][10]} The service then stores this data and answers analytic ad hoc queries on it with seconds of latency. It is a full-text indexing and retrieval database, including time series analysis capabilities^[6][7] and regular expression evaluation and text parsing.^[11]

It is offered as platform as a service (PaaS) as part of Microsoft Azure platform. The product was announced by Microsoft in 2018.

The development of the product began in 2014 as a grassroots incubation project in the Israeli R&D center of Microsoft,^[12] with the internal code name 'Kusto'^[9][7] (named after Jacques Cousteau, as a reference to "exploring the ocean of data"). The project aim was to address Azure services' needs for fast and scalable log and telemetry analytics.

In 2016 it became the backend big-data and analytics service for Application Insights Analytics.^[13]

The product was announced as a Public Preview product at the Microsoft Ignite 2018 conference,^[14] and was announced as a generally available at the Microsoft Ignite conference of February 2019.^[15]

In March 2021, "Kusto EngineV3", Azure Data Explorer's next generation storage and query engine, became generally available. It was designed to provide unparalleled performance for ingesting and querying telemetry, logs, and time series data.^[16]

Azure Data Explorer offers an optimized query language and visualizing options^[17] of its data with a SQL-like language called KQL (Kusto Query Language^[18][19][20]).^[7][8]

Azure Data Explorer can ingest 200 MB per second per node.^[14] Data Ingestion methods are pipelines and connectors to common services like Azure Event Grid or Azure Event Hub,^[21] or programmatic ingestion using SDKs.

Data visualization can be achieved using their native dashboard offering, or with tools like Power BI^[21][22] or Grafana.^[23][24]

Azure Data Explorer is a distributed database running on a cluster of compute nodes in Microsoft Azure. It is based on relational database management systems (RDBMS), supporting entities such as databases, tables, functions, and columns. It supports complex analytics query operators, such as calculated columns, searching and filtering on rows, group by-aggregates and joins.^[25]

The engine service exposes a relational data model: At the top level (cluster) there is a collection of databases, each database contains a collection of tables and stored functions. Each table defines a schema (ordered list of typed fields).

In Azure Data Explorer, unlike a typical relational database management systems (RDBMS), there are no constraints like key uniqueness, primary and foreign key.^[26] The necessary relationships are established at the query time.^[27] The data in Azure Data Explorer generally follows this pattern:^[28] Creating Database, Ingesting data, Query the database.