Ingest data into the Microsoft Sentinel SIEM and platform

A detection rule that hunts for repeated failed logons queries the SecurityEvent table, and it stays silent if your Windows events landed in WindowsEvent or Event instead. That one fact frames this page: getting data into Microsoft Sentinel means choosing a collection mechanism and, with it, the table that mechanism writes, because Sentinel content is written against table names. For workspace-backed ingestion, Sentinel stores records in tables of the Log Analytics workspace that Sentinel is enabled on.

This page covers how data gets into that workspace: after you onboard Sentinel, you use data connectors to start ingesting your data^[1], and the paths below also cover API-based ingestion. Once data is flowing, Microsoft Sentinel offers two retention tiers: the analytics tier, whose data is also mirrored to the data lake tier by default, and the data lake tier, which keeps data in the data lake only^[2] (see also Configure the Microsoft Sentinel SIEM and platform), and analytics rules query the collected data and generate alerts^[3] (see also Configure detections).

AMA-based collection, diagnostic settings, threat intelligence ingestion, and the Logs Ingestion API

The sections below take four ingestion paths in this order, from AMA-based collection, where Sentinel uses agents provided by Azure Monitor to collect data^[1], to the Logs Ingestion API:

1. AMA-based collection. The Azure Monitor Agent (AMA) runs on a Windows or Linux machine and follows a data collection rule (DCR), an Azure resource that says which events to collect, how to filter them, and where to send them. Microsoft Sentinel uses AMA for its Windows agent-based connectors^[4], and the Syslog via AMA and Common Event Format (CEF) via AMA connectors install AMA on Linux machines^[5].
2. Diagnostic settings. Azure resources emit platform logs, such as the Azure activity log, that diagnostic settings can send to a workspace. A diagnostic setting on the resource, or an Azure Policy assignment that creates those settings across a scope, streams the selected log categories to the workspace through a diagnostic settings-based connector^[6].
3. Threat intelligence ingestion. Threat indicators and related STIX objects (Structured Threat Information Expression, the open standard format for describing threat data) arrive through threat intelligence connectors, the upload API, or manual entry, and are stored in the ThreatIntelIndicators and ThreatIntelObjects tables^[7].
4. Logs Ingestion API. Your own application posts JSON to an endpoint, a DCR reshapes it, and the result lands in a table, typically a custom table whose name ends in _CL (Logs Ingestion API overview^[8]).

Two of the four paths share one policy object: AMA-based collection and the Logs Ingestion API are both DCR-based, so the DCR model taught in Data collection rules and associations below applies to both. Diagnostic settings and threat intelligence ingestion have their own configuration surfaces. The figure below maps each path to the tables it writes; diagnostic settings write AzureActivity for the Azure activity log^[9] and, for other resources, the table named in that resource's connector reference entry. AMA-based collection also writes Event when a generic Windows event DCR collects the logs: that route sends even Security log events to the Event table^[10], as Windows Security Events via AMA below explains.

The habit to take into every scenario: name the table the scenario's content needs, and the path and connector usually follow.

This section covers the steps every connector shares, from choosing it to proving it works; the later sections add each path's specifics. A data connector is the Sentinel component that configures collection for one kind of source, and connectors are delivered inside Content hub solutions, packages that also carry analytics rule templates, workbooks, and other content for that source.

Selecting a connector

Microsoft's enablement steps include reviewing the connector's Prerequisites and ensuring they're fulfilled^[11]; these four questions extend that check to the whole selection:

• Does it collect from your source? Each connector page states the source it supports and the prerequisites it needs, and some connectors document extra collection steps (enable a data connector^[11]).
• Does it write the table your content queries? This is the destination-table rule from the opening section: the Windows Security Events via AMA and Windows Forwarded Events connectors can carry the same Security log events into two different tables.
• Can you authorize it? Enabling a connector needs read and write permission on the Microsoft Sentinel workspace^[11], and individual connectors can require extra permissions documented per connector^[12]. Installing Content hub solutions needs the Microsoft Sentinel Contributor role at the resource group level^[13].
• Who supports it? Both Microsoft and other organizations author connectors, and the Supported by field on the connector's side panel names the support contact^[11]. A connector that ships through Content hub is not automatically maintained by Microsoft.

Enabling a connector

For Content hub connectors, the general enablement sequence is (enable a data connector^[11]):

1. Install the solution that contains the connector from Content hub.
2. Open the connector: in the Defender portal select Microsoft Sentinel > Configuration > Data connectors (in the Azure portal, Configuration > Data connectors), select the connector, and select Open connector page. If the connector is missing from the list, first check that its solution is installed.
3. Review and meet the Prerequisites listed on the page.
4. Follow the steps in the page's Configuration section, such as creating a DCR or launching a policy assignment.
5. Confirm data: the connector page shows a Data received graph and the connectivity status of each data type, and you can query the destination table in Advanced hunting (Defender portal) or Logs (Azure portal).

The figure below traces that sequence. Do not skip step 5: installing the solution in step 1 makes the connector available, but configuration in step 4 and confirmation in step 5 are still needed.

Two more points apply once a connector is enabled: a UEBA option on its page, and a default storage behavior. On a connector that supports User and Entity Behavior Analytics (UEBA, Sentinel's behavioral baselining of users, hosts, and other entities), Advanced options > Configure UEBA lets you toggle on the tables UEBA should analyze^[11]; connector connectivity alone does not prove that its tables are enabled for UEBA. And if your tenant is onboarded to the Microsoft Sentinel data lake, data from a newly enabled connector goes to the analytics tier and is mirrored to the data lake tier by default^[11]; managing those tiers and their retention is covered on the SIEM platform page.

Microsoft Sentinel in the Azure portal is supported until March 31, 2027^[11], after which Sentinel is available only in the Defender portal, so this page names the Defender portal path first.

The takeaway: installed is not connected, and connected is not proven until the expected table holds records.

This section covers the policy object behind AMA-based collection, because every AMA connector on this page creates one. A data collection rule (DCR) is an Azure resource that specifies the data to collect, the schema of the incoming data, any transformation to apply before storage, and the destination^[14]. Sentinel's agent-based connectors rely on two of its properties: DCRs are independent of both the workspace and the virtual machine, so one rule can be defined once and reused, and they filter at the source^[4], so the agent ingests only the events you asked for and leaves the rest behind.

How a DCR reaches a machine

A DCR does nothing until it is applied. A data collection rule association (DCRA) binds a rule to a machine, and the relationship is many-to-many: one DCR can serve many resources, and one resource can have up to 30 DCRs^[14]. When AMA is installed, it connects to Azure Monitor to retrieve the DCRs associated with it^[14], then collects and sends what those rules describe. The figure below shows one rule associated with two kinds of machine.

You can use the connector wizard instead of creating the association by hand. The connector page's +Add data collection rule wizard (create DCRs through the GUI^[4]) handles it in three configuration tabs before Review + create:

• Basics: the rule name, subscription, and resource group. The DCR does not have to sit in the same subscription or resource group as the machines, as long as everything is in the same tenant.
• Resources: the Azure VMs and Azure Arc-enabled servers (non-Azure machines given an Azure resource representation, explained in Machines outside Azure below) the rule applies to. Selecting them creates the associations, and the wizard installs AMA on any selected machine that lacks it.
• Collect: the events to collect, which differ per connector (event sets and XPath queries for Windows, facilities and severities for Syslog and CEF).

Because the rule and its targets are separate, scope collection by machine set: reuse one rule wherever the requirement is identical, and create a second rule for the subset of servers that needs different events, instead of editing the shared rule.

Machines outside Azure

An on-premises server, or a server in another cloud, is not an Azure resource, so a DCR cannot be associated with it directly. Such a machine must have Azure Arc installed and enabled before you enable the AMA-based connector^[4]; that covers physical Windows servers, on-premises virtual machines, and virtual machines in non-Azure clouds. The two agents do different jobs: the Azure Arc Connected Machine agent gives the server an Azure resource representation that AMA can be deployed to and a DCR associated with, and AMA still does the collecting. The older Log Analytics agent (also called MMA) was retired on 31 August 2024^[15], so it is never the answer for new collection.

Anatomy of a Windows Security Events DCR

The connector wizard builds the DCR for you, and the same rule can be created through the API^[4], which helps when you manage many rules. This request body follows that documented example, simplified to two Security log event IDs:

Request body: a DCR for Windows Security Events via AMA (simplified)

{
  "location": "eastus",
  "properties": {
    "dataSources": {
      "windowsEventLogs": [
        {
          "name": "eventLogsDataSource",
          "streams": ["Microsoft-SecurityEvent"],
          "xPathQueries": ["Security!*[System[(EventID=4624) or (EventID=4625)]]"]
        }
      ]
    },
    "destinations": {
      "logAnalytics": [
        {
          "workspaceResourceId": "/subscriptions/.../workspaces/centralTeamWorkspace",
          "name": "centralWorkspace"
        }
      ]
    },
    "dataFlows": [
      { "streams": ["Microsoft-SecurityEvent"], "destinations": ["centralWorkspace"] }
    ]
  }
}

Read the listing's three blocks against the definition above. dataSources.windowsEventLogs is what to collect: xPathQueries is the source-side filter, and Microsoft-SecurityEvent is the stream (the named data feed) it produces. destinations.logAnalytics is where: a workspace, given a local name. dataFlows connects the stream to that destination name. The ... inside workspaceResourceId marks the elided subscription and resource group path; replace it with the full resource ID before sending.

One name collision to clear up at first touch: Microsoft Defender for Endpoint also has custom data collection rules^[16], configured in the Defender portal under Settings > Endpoints > Rules > Custom Data Collection, which capture extra endpoint telemetry and send it to Sentinel. They are a Defender for Endpoint feature, not the Azure Monitor DCRs this page describes, and the automation configuration page covers them.

The takeaway: the DCR holds the policy, the association applies it, Azure Arc makes non-Azure servers eligible, and AMA does the collecting.

This section covers the choices on the Collect tab of the Windows Security Events via AMA connector, which decide both ingestion volume and audit coverage. The connector writes to the SecurityEvent table, which many built-in Windows security analytics rules query^[17], and it offers four event sets:

Event set	What it collects	Choose it when
All events	The full, unfiltered Security log plus the AppLocker logs	You need every Security log event and can accept the volume
Common	A curated audit set that keeps a full user audit trail, including both sign-in (4624) and sign-out (4634) events	You need audit-grade coverage at reduced volume
Minimal	A small set of events that might indicate a breach, such as successful and failed logons (4624, 4625) and process creation (4688), without sign-out (4634)	You need breach signals at the lowest volume and no full audit trail
Custom	The events your own XPath queries select	You need specific event IDs or conditions

The event set reference^[17] lists every event ID in Common and Minimal. Distinguish these two event-set limitations. First, All events is not all Windows logs: it covers only the Security and AppLocker channels and does not include Application, System, or Setup^[17]. Second, Minimal is not a smaller Common: it drops sign-out event 4634 because that event matters for auditing but not for breach detection, so a requirement for complete session auditing rules Minimal out.

Generic Windows event DCRs write a different table

Azure Monitor can also collect Windows events through a generic DCR with the Windows Event Logs data source. If you select the Security log there, the events go to the Event table^[10], alongside events from logs such as System and Application, not to SecurityEvent. Collecting the same event IDs through that route does not satisfy content written for SecurityEvent; the destination-table rule from the opening section decides the connector.

Custom XPath filters

An XPath query here is a filter expression over the XML form of a Windows event, evaluated by the agent before anything is sent. DCR entries take the form LogName!XPathQuery: the channel name, an exclamation mark, then the XPath. The Azure Monitor documentation^[10] gives examples such as these:

XPath syntax examples from the Azure Monitor Windows events documentation

Security!*[System[EventID=4625]]
Security!*[System[(EventID=4648)]] and *[EventData[Data[@Name='ProcessName']='C:\Windows\System32\consent.exe']]
System!*[System[(Level=1 or Level=2 or Level=3) and (EventID != 6)]]

The first line selects failed logon event 4625 from the Security channel; the second selects event 4648 only when the process is consent.exe; the third selects critical, error, and warning System events except event ID 6. The syntax is the same in this connector's Custom set, whose Collect step links to these XPath rules^[4], and in a generic Windows event DCR. For System and Application events, this page uses the generic DCR route, which writes them to the Event table^[10]. The field expects Windows Event Log XPath, not KQL, so a KQL where clause typed there is invalid. Filtering here also keeps unwanted events from ever being ingested, which saves ingestion cost^[4], whereas a KQL filter at query time only hides records you have already paid to store.

Three constraints apply to this collection configuration:

• AMA supports XPath version 1.0 only^[4], and functions such as starts-with and contains are not supported^[10].
• The connector accepts up to 20 expressions in a single box and up to 100 boxes in a rule^[4].
• Analytic and Debug channels cannot be collected^[10], because Windows does not allow the event subscription AMA uses on those channel types.

Testing an XPath query locally

Before saving a custom set, test the expression on a Windows machine with Get-WinEvent. The value of -LogName is the part before the exclamation mark, and -FilterXPath takes the rest (testing XPath with Get-WinEvent^[10]):

PowerShell: test the XPath part of Security!*[System[EventID=4625]]

$XPath = '*[System[EventID=4625]]'
Get-WinEvent -LogName 'Security' -FilterXPath $XPath

Returned events mean the query is valid. The message "No events were found that match the specified selection criteria" means the query may be valid but nothing on this machine matches, while "The specified query is invalid" means a syntax error (interpreting the result^[4]). One trap remains: Get-WinEvent supports up to 23 expressions, but DCRs support up to 20^[10]. That 20 is the per-box limit stated above, not a cap per rule, so a local test that passes with 21 to 23 expressions in one query does not prove a DCR XPath box will accept it.

The takeaway: choose the event set by audit requirement, keep Security content on this connector so it lands in SecurityEvent, and validate custom XPath locally, keeping each XPath box within 20 expressions.

This section covers the second Windows option: collecting from servers that already forward their events, instead of installing AMA on each one. Windows Event Forwarding (WEF) is Windows' built-in subscription mechanism: source computers forward selected events to a central server running Windows Event Collection, the Windows Event Collector (WEC), which stores them in its Forwarded Events log. Sentinel then collects from the collector alone.

Planning the collector

The Windows Forwarded Events connector's prerequisites^[4] set the plan:

• WEC enabled and running, with AMA installed on the WEC machine. The collector is the only machine Sentinel needs an agent on; installing AMA on the sources instead leaves the collector's events unsent.
• Azure Arc on a non-Azure collector. The Arc rule from Data collection rules and associations applies to the collector like any other server.
• ASIM parsers recommended. The Advanced Security Information Model (ASIM) is Sentinel's normalization layer, a set of parsers that present data from different sources in one common schema; Microsoft recommends deploying the ASIM WindowsEvent parsers^[4] for full normalization support.

The table changes: WindowsEvent, not SecurityEvent

Events collected through WEF are written to the WindowsEvent table, including events forwarded from the Windows Security log; they are not written to SecurityEvent^[4]. Many built-in Windows Security Events rules, including some near-real-time rules, query SecurityEvent and don't match data stored only in WindowsEvent^[4]. Microsoft names three ways forward:

1. Collect directly with Windows Security Events via AMA when you need the built-in SecurityEvent rules unchanged.
2. Adapt the rule queries to WindowsEvent.
3. Use supported ASIM parsers so normalized content can read the forwarded events.

Adapting a query is more than swapping a table name: the two tables have different schemas, so every field reference must be checked as well.

Configuring collection in two stages

Once the WEC subscriptions deliver events, open the Windows Forwarded Events connector, create a DCR, associate it with the collector, and choose All events or a Custom XPath selection on the Collect tab (the DCR wizard^[4]). Delivery now has two independent stages, which the figure below separates. Stage 1 is WEF delivering source events to the collector; stage 2 is AMA delivering the collector's forwarded events to the workspace. When events are visible in the collector's Forwarded Events log but absent from WindowsEvent, stage 1 is working, and stage 2 (the agent, the DCR, or its association) is the place to look.

The takeaway: WEF puts one agent on the collector and lands everything in WindowsEvent, so plan the rule and parser consequences before choosing it over the direct connector.

This section covers how Linux and appliance logs reach Sentinel, starting with the two formats involved. Syslog is a standard protocol for sending log messages between devices and applications; typical fields include a priority, a timestamp, a hostname, an application name, a process ID, and message text, with availability depending on the format and sender, and messages can travel over UDP, TCP, or TLS. AMA supports Syslog formatted according to RFC 3164 and RFC 5424^[5]. Common Event Format (CEF) is a vendor-neutral extension of Syslog built for SIEM use: a standard header (device vendor, product, version, event class, severity, and event ID) followed by extensions such as source and destination IP addresses, user names, and actions (what Syslog and CEF are^[5]).

Two topologies

The Syslog via AMA and CEF via AMA connectors^[5] install AMA on a Linux machine in one of two roles:

• Single machine. A Linux VM that produces its own Syslog runs AMA itself.
• Log forwarder. A dedicated Linux VM, on-premises, in Azure, or in another cloud, receives messages from appliances and other machines and runs AMA on their behalf. Firewalls and other appliances that only export Syslog use this topology, because AMA runs on the forwarder, never on the appliance.

The hops on a log forwarder

The figure below traces a message through a forwarder:

1. The appliance sends Syslog or CEF to the Syslog daemon on the forwarder (rsyslog or syslog-ng), on TCP or UDP port 514 or another port you prefer^[5]. Port 514 is the convention, not a requirement; if you change it, the daemon's port must match the sender's.
2. The daemon passes messages to AMA: AMA 1.28.11 and later receives them on TCP port 28330, and earlier versions use a Unix domain socket^[5]. Port 28330 is an internal hop on the forwarder, so appliances are never pointed at it.
3. AMA applies the DCR and sends the result to the workspace, where CEF messages land in CommonSecurityLog and plain Syslog in Syslog^[5].

Every Log Analytics table has a TimeGenerated datetime column^[18] (Creating custom log tables below covers it). On a forwarder, TimeGenerated reflects when the forwarder processed the message, while EventTime comes from the Syslog header^[5], which, when it carries no time zone, is converted using the forwarder's offset, so the two fields can disagree when the device and the forwarder sit in different time zones.

Prerequisites

The connector article's prerequisites^[19] for a forwarder include, among others:

• A designated Linux VM, with the Azure Arc Connected Machine agent installed if it is not an Azure VM.
• Python 2.7 or 3, and either the syslog-ng or the rsyslog daemon enabled.
• Sources configured to send to the forwarder's daemon instead of their own local daemon.
• Azure roles: Virtual Machine Contributor or Azure Connected Machine Resource Administrator to install the agent, a role that includes Microsoft.Resources/deployments/* to deploy templates, and Monitoring Contributor to create or edit DCRs.

If devices send over TLS, for example because the forwarder runs in the cloud, you configure the Syslog daemon to communicate in TLS^[19]. AMA's own connection to Azure does not protect the appliance-to-forwarder leg; that leg is daemon configuration.

Setup sequence

1. Install the Syslog or Common Event Format solution from Content hub to get the matching connector (setup process^[5]).
2. On the connector page, select +Create data collection rule, pick the forwarder on the Resources tab, and set facilities and severities on the Collect tab; creating the rule installs AMA on the machines you selected^[19]. The API alternative requires you to install AMA manually before creating the DCR^[5], then create the association yourself.
3. On the forwarder, run the installation script Forwarder_AMA_installer.py, copying its command from the connector page. The script configures the daemon, restarts it, and opens port 514 for both UDP and TCP^[19]; the listening port can be changed in /etc/rsyslog.conf or /etc/syslog-ng/syslog-ng.conf.
4. Configure each appliance to send to the forwarder's listener.

Step 3 configures the daemon to receive messages from other machines: AMA and a DCR alone do not ensure a remote listener, so verify the daemon's listening configuration rather than recreating the DCR.

The takeaway: appliances talk to the daemon, the daemon talks to AMA, and AMA talks to Azure, so configure and troubleshoot each hop separately.

This section covers what the DCR of a Syslog or CEF connector selects, and how a forwarder that handles both formats avoids storing a message twice.

Facilities and minimum log levels

A Syslog facility is a category label the sender attaches to each message (such as auth, daemon, or local4), and the log level is its severity. On the connector's Collect tab you pick a minimum log level for each facility, and selecting a level collects it and every more severe level: choosing LOG_ERR collects LOG_ERR, LOG_CRIT, LOG_ALERT, and LOG_EMERG^[19]. Less severe messages from that facility, such as informational ones, are excluded, so check that the minimum you pick still keeps the events your detections need. The portal offers only this minimum-level choice; creating the DCR through the API lets you filter by specific log levels^[5].

In the DCR itself, each Syslog data source names its stream, its facilities, and its levels. This excerpt follows Microsoft's Syslog and CEF streams in the same DCR^[19] example:

DCR excerpt: one CEF stream and one Syslog stream on separate facilities

"dataSources": {
  "syslog": [
    {
      "name": "CEFStream1",
      "streams": ["Microsoft-CommonSecurityLog"],
      "facilityNames": ["authpriv", "mark"],
      "logLevels": ["Info", "Notice", "Warning", "Error", "Critical", "Alert", "Emergency"]
    },
    ...
    {
      "name": "SyslogStream3",
      "streams": ["Microsoft-Syslog"],
      "facilityNames": ["kern", "local0", "local5", "news"],
      "logLevels": ["Critical", "Alert", "Emergency"]
    }
    ...
  ]
}

The ... lines mark the other streams of Microsoft's example, which are omitted here; this is a fragment of a full DCR, not a complete request body. The streams value decides the table: Microsoft-CommonSecurityLog for CEF and Microsoft-Syslog for plain Syslog. The two entries list disjoint facilityNames, and that separation is what prevents duplicates. The logLevels values name each severity in words: the portal's LOG_ERR, LOG_CRIT, LOG_ALERT, and LOG_EMERG appear in DCR JSON as Error, Critical, Alert, and Emergency (facility and log level examples^[19]).

Preventing duplicate ingestion

If CEF and plain Syslog share a facility, the same message can be ingested into both CommonSecurityLog and Syslog^[5]. Enabling every facility on both connectors for completeness causes exactly that. Microsoft documents two fixes:

1. If the source can change its facility: send CEF and plain Syslog on different facilities, and configure each DCR stream to collect only its own facilities.
2. If it cannot: add an ingestion-time transformation to the Syslog stream that drops CEF messages, and keep the CEF stream so the data is still stored once.

KQL transformation on the Syslog stream: drop CEF messages

source
| where ProcessName !contains "CEF" and SyslogMessage !contains "CEF:0"

The transformation runs in the DCR before storage (source is the incoming stream), and it checks two fields because, starting with AMA 1.41, ProcessName might not reliably contain "CEF" for vendors whose headers don't comply with RFC 3164 or RFC 5424^[5]. Filtering on ProcessName alone can let those vendors' CEF messages through into Syslog.

The takeaway: facilities and minimum levels decide what is kept, and disjoint facilities or a Syslog-stream transformation help prevent this CEF/Syslog duplication for selected messages.

This section covers proving a Syslog or CEF pipeline end to end. Work the hops in order, because each check proves only its own hop (test the connector^[19]):

1. Run netstat -lnptv on the forwarder and confirm that rsyslog or syslog-ng listens on port 514.
2. Capture traffic with sudo tcpdump -i any port 514 or 28330 -A -vv & to watch messages arrive at the daemon and pass to AMA.
3. Send a CEF-formatted test message to the local daemon, for example with the logger command below.
4. Query the destination table. Logs can take up to 20 minutes to appear^[19] in the workspace.

Bash: send a mock CEF test message to the local daemon

logger -p local4.warn -P 514 -n 127.0.0.1 --rfc3164 -t CEF "0|Mock-test|MOCK|common=event-format-test|end|TRAFFIC|3|rt=Nov 04 2018 07:15:46 GMT"

KQL: confirm the test message reached CommonSecurityLog

CommonSecurityLog
| where TimeGenerated > ago(1d)
| where DeviceProduct == "MOCK"

In the logger line, -t CEF plus the quoted text form the CEF header CEF:0|Mock-test|MOCK|common=event-format-test|end|TRAFFIC|3| before the rt extension (Microsoft also shows an nc (netcat) test^[19]): version 0 (the CEF version, which comes before the header fields^[20] in the documented order CEF:Version|Device Vendor|Device Product|Device Version|...|Severity|[Extension], and is not the device version listed in Syslog and CEF via AMA above), then vendor Mock-test and product MOCK, the first two header fields, stored in DeviceVendor and DeviceProduct. In that order Severity^[20] is the last header field, so the next four values run from Device Version through Severity: common=event-format-test is a mock device version despite its =, only the rt= part after the last pipe is an extension, and 3 is the CEF Severity, not the syslog Warning level that -p local4.warn sets^[19]. The query filters on DeviceProduct, and ago(1d) still finds the message, because the 2018 rt value is only sample content and TimeGenerated is the forwarder's processing time^[5] (Syslog and CEF via AMA above). For a real appliance, filter on its vendor and product, such as DeviceVendor == "Cisco" and DeviceProduct == "ASA" in Microsoft's Cisco ASA example^[19].

A daemon listening on 514 says nothing about AMA, and traffic captured on port 514 proves only that the message reached the forwarder. Only the table query proves that the DCR selected the message, AMA parsed it, and the workspace stored it. If the test message never arrives, the connector article^[19] points to a dedicated troubleshooting guide.

The takeaway: validate hop by hop, and count a CEF source as onboarded only when the expected DeviceVendor and DeviceProduct rows appear in CommonSecurityLog.

This section covers the diagnostic settings path, used for logs that the Azure platform produces about your resources. The Azure activity log records subscription-level events such as resources being created, changed, or deleted, and resource logs record operations inside an Azure resource; neither comes from a guest operating system, so installing AMA on a VM never collects them. A diagnostic setting on the source sends selected log and metric categories to a destination, and for Sentinel that destination is the Log Analytics workspace that Sentinel is enabled on.

Two connector types

Diagnostic settings-based connectors come in two kinds^[6], and the figure below shows each kind and what it covers:

• Standalone connectors have you configure one resource's own diagnostic settings.
• Connectors managed by Azure Policy apply a single diagnostic settings configuration to every resource of one type within a scope.

Standalone: one resource at a time

From the connector page, follow the link to the resource, then (standalone procedure^[6]):

1. Select Diagnostic settings > + Add diagnostic setting, and name the setting.
2. Select Send to Log Analytics, then choose the subscription and the Log Analytics Workspace where Microsoft Sentinel resides.
3. Select the log and metric categories to collect, following the recommendations in the connector's reference entry, and select Save.

The destination is the part that goes wrong. A diagnostic setting that sends only to a storage account, or to another workspace, is a working setting that does not reach the intended Sentinel workspace through that setting; checking that the correct workspace is selected^[9] is the first troubleshooting step when logs don't appear.

Azure Policy: a whole scope, now and later

For a policy-managed connector, select Launch Azure Policy Assignment wizard on the connector page (policy-managed procedure^[6]):

1. Basics: choose the scope, a subscription and optionally a resource group.
2. Parameters: clear Only show parameters that require input, choose the Sentinel workspace from the Log Analytics workspace list, and leave True on each log type you want to ingest.
3. Remediation: the policy applies to resources added in the future; to apply it to existing resources as well, select Create a remediation task.
4. Review + create: select Create.

Assignment and remediation cover different parts of the fleet: the assignment configures resources as they are added, and the remediation task back-fills the ones that already exist. Assigning without remediation does not back-fill diagnostic settings on existing noncompliant resources. Applying a log streaming policy this way requires the Owner role for the policy assignment scope^[6], on top of read and write permission on the workspace.

The Azure Activity connector

The Azure Activity connector now uses the diagnostic settings pipeline^[6]. If a subscription is still connected through the legacy method, disconnect it from that method first, using the connector page's instructions, before setting up the new connection, so the two collection paths are not both active. Activity data lands in the AzureActivity table, which Microsoft's troubleshooting article queries to verify the transfer^[9], and Azure Activity is one of the free Microsoft Sentinel data sources^[21].

Reading the connection status

For policy-managed connectors, the status shows connected only if data has been ingested at some point in the past 14 days^[6]; after 14 days without data it shows disconnected, and it returns to connected as soon as data arrives. A quiet source can therefore look disconnected with its configuration intact, so check source activity and the table before recreating anything.

The takeaway: platform logs need a diagnostic setting pointed at the right workspace, Azure Policy scales that setting to a scope, and a remediation task covers the resources that already exist.

This section covers how threat intelligence gets into Sentinel, starting with what it is. A threat indicator, also called an indicator of compromise (IOC), associates an observed artifact such as a URL, a file hash, or an IP address with known threat activity such as phishing, botnets, or malware^[7]. Sentinel also stores richer STIX objects: threat actors, attack patterns, identities (such as victim organizations), and the relationships between them. STIX is the data format; TAXII (Trusted Automated Exchange of Intelligence Information) is the protocol that servers use to exchange it, and the combination of the two is the most widely adopted standard for transmitting threat intelligence^[7].

Three routes in

Threat intelligence arrives by one of three routes, and the figure below shows each one and where ingestion rules apply (ordered rules that filter or enrich objects arriving from connectors, explained in Managing ingested threat intelligence below):

• Pull, through a data connector. Sentinel fetches intelligence from a feed: a TAXII server's collection or Microsoft's own threat intelligence.
• Push, through the upload API. Your threat intelligence platform (TIP) or custom application sends objects to Sentinel.
• Manual entry. An analyst creates an object in the threat intelligence management interface.

The connectors below are available in Content hub as part of the Threat Intelligence solution^[7], and installing it needs Microsoft Sentinel Contributor at the resource group level^[13].

Pull: data connectors

Microsoft Defender Threat Intelligence connectors. The standard connector brings public indicators and open-source intelligence (OSINT), and the premium connector adds Microsoft's curated indicators and Microsoft-enriched OSINT^[7]. Both put the intelligence in your workspace for monitoring, alerting, and hunting. Do not confuse them with the free Defender Threat Intelligence matching analytics rule: with matching analytics, only indicators that match the rule are ingested into your environment^[7], so it is not a way to build a full indicator inventory for other hunts.

Threat Intelligence - TAXII connector. It is a built-in TAXII client that pulls threat indicators from TAXII 2.0 or 2.1 servers^[13]. On the connector page you enter (enable the TAXII connector^[13]):

• a Friendly name for the server collection;
• the API root URL and the Collection ID, which the provider usually documents; some providers advertise only a discovery endpoint, which you browse to find the API root, because a discovery URL alone does not identify a collection;
• a Username and Password, if the server needs them;
• the group of indicators to import and the polling frequency.

Repeat for each collection you want. A few servers, such as FS-ISAC, require you to allowlist the Microsoft Sentinel TAXII client's IP addresses^[13]; most do not. A separate Threat Intelligence - TAXII Export connector sends threat data out to TAXII 2.1 servers^[13], so keep import and export apart.

Push: the upload API

The threat intelligence upload API, in preview, ingests STIX objects into Sentinel without the need for a data connector^[22], scoped to one workspace. Its setup (configure the upload API^[22]):

1. Register a Microsoft Entra application and record its application (client) ID.
2. Generate a client secret for it.
3. Assign the application the Microsoft Sentinel Contributor role at the workspace level.
4. Configure the TIP or custom application with the client ID, a Microsoft Entra access token obtained with OAuth 2.0, and the Sentinel workspace ID.

The legacy Threat Intelligence Platform data connector also accepts pushed intelligence, but it uses the Microsoft Graph Security tiIndicators API, is limited to indicators, and is being deprecated^[7], so new integrations that carry threat actors or relationships use the upload API. Mind the name collision: your TIP is the product that sends intelligence, while the Threat Intelligence Platform data connector is a legacy Sentinel receiver that shares its name. Pushing needs no TAXII server: the upload API is itself the receiving end.

Manual: a TI object

A finding from your own investigation needs no feed at all. In the threat intelligence management interface, select Add new > TI object, choose the Object type, complete the required fields, and select Add (create threat intelligence^[23]).

Where it lands

All three routes store data in the same tables: ThreatIntelIndicators for indicators and ThreatIntelObjects for the other STIX objects^[7]. Sentinel stopped ingesting into the legacy ThreatIntelligenceIndicator table after July 31, 2025^[7], so new queries, analytics, workbooks, and validation checks target the new tables.

Ingesting is not detecting. Stored indicators raise alerts only when an analytics rule compares them with collected events, such as the threat intelligence rule templates or Microsoft's matching analytics rule, and those rules are covered on the detections page. Each template lists the data sources it requires^[7], so an indicator feed with no matching activity table produces no alerts.

The takeaway: pull with connectors, push with the upload API, enter local findings by hand, and enable detection as a separate step.

This section covers what you can do to threat intelligence on its way in and after it is stored, and how to count it correctly.

Ingestion rules

Threat intelligence ingestion rules filter or enrich objects before they are delivered to the workspace, and they apply to data connectors only, not to intelligence added through the upload API or created manually^[7]. Their evaluation model (tips for ingestion rules^[7]):

• Rules apply in order, and each incoming object is processed by every rule until a Delete action is taken.
• Delete skips that object, removing it from the ingestion pipeline; any previously ingested versions of the object are not affected.
• An object that no rule acts on is ingested from the source as is.
• New and edited rules take up to 15 minutes to take effect.

Typical uses include filtering out old, low-confidence intelligence, extending the Valid until date of high-fidelity indicators by 30 days, and adding tags (ingestion rule use cases^[7]). Because Delete acts only on incoming objects, it cannot purge an indicator that is already stored.

These rules are not the DCR transformations taught earlier on this page: a DCR transformation is a KQL query over incoming log records, whereas a threat intelligence ingestion rule is an ordered rule list over STIX objects arriving from connectors.

Relationships, tags, and sensitivity

Three curation tools record three different kinds of meaning (curate threat intelligence^[7]):

• Relationships are typed links between objects, built with the relationship builder: a threat actor Uses an attack pattern, an indicator is Attributed to a threat actor, and an attack pattern Targets an identity.
• Tags are free-form labels for grouping and finding objects, typically by incident. When an object represents a known actor or campaign, create a relationship instead, because a tag records no typed connection.
• Traffic Light Protocol (TLP) marks who may receive an object:

TLP color	Sharing allowed
White	Freely and publicly, without restriction
Green	With peers and partner organizations in the community, but not publicly
Amber	With members of the organization, but not publicly
Red	Only within the specific group or meeting where it was originally disclosed

Set TLP in the interface when you create or edit an object; through the upload API it is set in object_marking_refs by choosing one of four marking-definition object GUIDs^[7]. For intelligence from partner sources such as TIP and TAXII feeds, only specific fields are editable, including tags, Expiration date, Confidence, and Revoked^[23].

Microsoft also enriches IP and domain indicators with GeoLocation and WhoIs data (public preview)^[7]: an IP's organization and country or region, or a domain's registrar and record creation data. These attributes describe the indicator's infrastructure; they are context, not evidence that anything in your environment communicated with it.

Counting current indicators

Raw row counts overstate the inventory. When an indicator is created, updated, or deleted, Sentinel creates a new entry in the tables, and it automatically reingests all threat intelligence every seven to 10 days^[7]. The management interface shows only the most current version, choosing for each Id the row with the newest TimeGenerated. Do the same in KQL, then filter on the IsActive and IsDeleted columns^[24]:

KQL: count current, active indicators by observable key

ThreatIntelIndicators
| summarize arg_max(TimeGenerated, *) by Id        // newest version of each indicator
| where IsActive == true and IsDeleted == false    // drop inactive and deleted ones
| summarize ActiveIndicators = count() by ObservableKey

An indicator's Pattern is its detection pattern, which can be a STIX pattern^[24], such as [ipv4-addr:value = '172.29.6.7'] in Microsoft's upload API example^[25]. ObservableKey is the entire left-hand side of an equality comparison from the pattern^[24], ipv4-addr:value in that example, so the result counts current indicators per observable key. The Id column is a concatenation of the base64-encoded SourceSystem, three dashes, and the STIX ID^[7], so the same STIX object from two sources counts as two indicators. Run the query over a time range longer than the reingestion cycle so every current indicator has at least one row in range; the reingested rows themselves are charged at standard ingestion pricing for paid tables^[7].

The takeaway: ingestion rules shape connector input only, relationships and TLP carry meaning that tags don't, and the current inventory is the latest version per Id.

This section covers custom tables, the destination for data whose shape no built-in table matches. Azure tables have predefined schemas; to store log data in a different schema, you use DCRs to collect, transform, and send it to a custom table^[18] in the workspace. Managing a table needs Microsoft.OperationalInsights/workspaces/* permission on the workspace, as provided by Log Analytics Contributor, for example^[18].

The rules every custom table follows

• The name ends in _CL. The Azure portal adds the suffix automatically; with any other method you add it yourself^[18], and the table name in the DCR's data flow must match the workspace table name. Table names are used for billing, so keep sensitive information out of them.
• It has a TimeGenerated datetime column. Every Log Analytics table must have one; if it is missing, Azure Monitor adds it to the transformation in your DCR^[18]. Map a real source timestamp into it where one exists, so time-based queries reflect when the event happened.
• It has a table plan. You choose Analytics (the default), Basic, or Auxiliary / Lake at creation^[18], and all DCR-based custom tables support all plans^[26]. Choose by how the data will be used, not by its format: Analytics is suited to real-time detection, and Auxiliary logs cannot trigger alerts^[27], so prefer Analytics for data that feeds real-time detections. After creation, you change the plan in the Azure portal from the Log Analytics workspace's Tables list: select Manage table, then a value from the Table plan dropdown on the table configuration screen^[26], and plan changes are limited to one switch per table per week^[26]. The analytics and data lake tiers named in Data connectors: from selection to confirmed data above are set from a different control in a different portal, Microsoft Sentinel > Configuration > Tables in the Defender portal^[28]; the rest of this section speaks only of plans (see also Configure the Microsoft Sentinel SIEM and platform).
• Auxiliary / Lake timestamps are strict. These plans support TimeGenerated only in ISO 8601 format with six decimal places (microsecond precision)^[18], such as 2026-10-21T08:15:30.123456Z, so declaring a datetime column does not by itself make an incoming timestamp valid.

Creating a table in the portal

The portal workflow (create a custom table^[18]) builds the table and its ingestion DCR together:

1. In the Log Analytics workspace, select Tables > Create, and enter a name (without _CL) and an optional description.
2. Select the Table plan.
3. Select an existing data collection rule or create a new one, and select a data collection endpoint (DCE, a separate Azure resource that provides an ingestion endpoint, explained in Sending records with the Logs Ingestion API below).
4. Provide a sample of the source data as JSON. If the sample has no TimeGenerated column, the portal tells you it will create a transformation that adds one.
5. Optionally open the Transformation editor, write the KQL transformation, and select Run to preview the results.
6. Select Apply, review the new table's schema, and select Create.

The figure below condenses those six steps into five stages; its last box is a created table and route, not ingested data. The transformation in step 5 is a KQL query that runs against each incoming record, and Azure Monitor Logs stores the query's results^[18]. That makes it the place to drop sensitive fields: a saved query or function applied at read time changes what you see, while the stored records keep every field. Creating the table and its DCR prepares the route but sends nothing; records arrive only once an authorized sender uses the route, which the next section covers.

Column names

Custom column names follow rules stricter than general Kusto identifier rules^[18]:

Rule	Example
Start with an ASCII letter, then use only letters, digits, or underscores	SourceIp is valid; source.ip and source-ip are not
Be 2 to 45 characters long	Severity
Avoid system and reserved names, including id, TenantId, Title, Type, UniqueId, _ResourceId, and _SubscriptionId	Map a vendor's Type field to SourceType
On Auxiliary tables, avoid names that differ only by case	Ingestion drops data for duplicates such as Status and status

Kusto can reference unusual property names by quoting them in a query, but the custom table schema accepts only letters, digits, and underscores^[18], so quoting does not rescue a dotted source field name. Column names are case sensitive only for Analytics and Basic tables^[18].

Extending built-in Azure tables

Built-in Azure tables follow different rules. You can't delete Azure tables, and in Azure tables you can only add and delete custom columns^[18], never remove the predefined ones; custom column names in Azure tables must end in _CF^[18], as in Classification_CF. When the schema you need differs from a built-in one, create a custom table instead of trying to reshape the Azure table.

Changing a table's schema

Two schema-change traps follow:

• Senders are not updated for you. Whenever you change a table schema, update the data collection rules that send data to the table; Azure Monitor doesn't update them automatically^[18].
• The CLI replaces the column list. With az monitor log-analytics workspace table update, the --columns argument replaces the custom column set rather than adding to it^[18]; any custom column you omit is deleted.

The listing below applies the second trap to an Azure table that already has a custom column Custom1_CF and needs a second one.

Azure CLI: add Custom2_CF while keeping Custom1_CF

az monitor log-analytics workspace table update \
  --resource-group "$resourceGroupName" \
  --workspace-name "$workspaceName" \
  --name "$tableName" \
  --columns Custom1_CF=string Custom2_CF=string

Passing only Custom2_CF=string would delete Custom1_CF, because --columns is the complete replacement set.

The takeaway: _CL, TimeGenerated, and a plan chosen by use define a custom table, and every schema change has to be carried into its DCRs.

This section covers the sender side of a custom table: how an application delivers records through a DCR. The Logs Ingestion API accepts data from any application that can make a REST API call^[8]; the call names a DCR, which holds the target table and workspace, and carries the credentials of an app registration (a Microsoft Entra application, the kind the upload API also uses) that has access to that DCR. The JSON has to match the structure the DCR expects, not necessarily the table, because the DCR can include a transformation to fit the table, and you can change the target table and workspace by modifying the DCR without changing the API call^[8]. The figure below traces one request, which goes to an endpoint specified by the DCR, or to a DCE if you're using private link^[8] or the DCR has no logs ingestion endpoint of its own (The endpoint: DCR or DCE below explains both cases).

The components

Three things must exist before the first call (configuration^[8]):

• An app registration with a secret. It authenticates the call, and it must be granted the Monitoring Metrics Publisher role on the DCR through the DCR's Access Control (IAM) page.
• The destination table. It must exist before data is sent: the API supports custom tables and selected Azure tables, and custom table names must have the _CL suffix^[8].
• A DCR. If you created the custom table in the portal, the DCR was created with it; otherwise you create it yourself.

The Logs Ingestion API replaces the legacy HTTP Data Collector API: it uses OAuth-based authentication, which is more secure than workspace keys, and DCR-governed schema control and transformations^[14].

The endpoint: DCR or DCE

The request goes to one of two endpoint types (endpoint^[8]):

• The DCR's own logs ingestion endpoint. A DCR created for direct ingestion, with kind set to Direct, generates a logsIngestion endpoint. You retrieve it from the DCR's JSON view in the Azure portal, switching to the latest API version if the endpoints aren't displayed. No separate resource is needed.
• A data collection endpoint (DCE). A DCE is a separate Azure resource that provides an ingestion endpoint. You need one only when connecting to the workspace through private link, or when the DCR has no logs ingestion endpoint, as with older DCRs or DCRs created without the Direct kind.

A table built in the portal (Creating custom log tables) has you select a DCE for its DCR^[18], so that route can use the DCE; a DCR with its own endpoints can also use a DCE, and you choose which one each client calls^[8].

The logsIngestion property was added on March 31, 2024, and endpoints can't be added to an existing DCR^[8]: to give an older route its own endpoint, create a new DCR. The workspace resource ID is never the endpoint; it is the DCR's destination.

Anatomy of an ingestion DCR

This request body creates a Direct DCR that routes a custom stream into a custom table, following Microsoft's create a custom table^[18] example:

Request body: a Direct DCR for a custom table (column list shortened)

{
  "location": "<Location>",
  "kind": "Direct",
  "properties": {
    "streamDeclarations": {
      "Custom-<TableName>_CL": {
        "columns": [
          { "name": "TimeGenerated", "type": "datetime" },
          { "name": "StringProperty", "type": "string" }
          ...
        ]
      }
    },
    "destinations": {
      "logAnalytics": [
        {
          "workspaceResourceId": "/subscriptions/<SubscriptionId>/resourceGroups/<ResourceGroupName>/providers/Microsoft.OperationalInsights/workspaces/<WorkspaceName>",
          "name": "<WorkspaceName>"
        }
      ]
    },
    "dataFlows": [
      {
        "streams": ["Custom-<TableName>_CL"],
        "transformKql": "source",
        "destinations": ["<WorkspaceName>"],
        "outputStream": "Custom-<TableName>_CL"
      }
    ]
  }
}

The ... line marks the further columns of Microsoft's example that are omitted here; remove it before you submit the body, or replace it with your real column entries and add a comma after the StringProperty object to separate the entries. Each key has one job (DCR properties^[8]):

Key	What it does
location	The DCR's region, which must match the region of the workspace, and of the DCE if you use one
kind	Direct gives the DCR its own logs ingestion endpoint
streamDeclarations	The columns of the incoming data, under a stream name
destinations.logAnalytics	The workspace, given a local name that the data flow refers to
transformKql	The KQL transformation; source passes the data on unchanged when it already matches the table
outputStream	The destination table: Custom-<table-name> for a custom table, Microsoft-<table-name> for a built-in one

The sender targets the input stream (the streamDeclarations key), and outputStream picks the table; in the listing both are Custom-<TableName>_CL, in two roles. Place the DCR in the workspace's region, not near the sender: use the same region for the DCR, the Log Analytics workspace, and the DCE if you use one^[8].

The call

The sender makes an HTTPS POST to a URI built from the endpoint, the DCR's immutable ID, and the stream name (REST API call^[8]):

URI format for a Logs Ingestion API call

{endpoint}/dataCollectionRules/{dcrImmutableId}/streams/{streamName}?api-version={apiVersion}

{streamName} is that streamDeclarations key.

Client libraries for .NET, Go, Java, JavaScript, and Python wrap the same call and require the same components^[8].

The takeaway: an authorized app posts JSON to the DCR's endpoint, the DCR declares, transforms, and routes it, and the table must already exist in the same region.

These illustrative scenarios practice the choices this page teaches, from the Windows Security Events via AMA event sets^[17] to a Syslog DCR's minimum log level, with the reasoning that separates the right option from the tempting one. Three questions recur in these scenarios: which path the source needs, which table the content queries (for example, Syslog messages appear in the Syslog table and CEF messages in CommonSecurityLog^[1]), and which stage of a pipeline is failing. Every row revisits a documented behavior that an earlier section explains, such as the All events set not including the Application, System, or Setup logs^[17].

Scenario	Sentinel or Azure Monitor configuration to use	Tempting wrong answer, and why it fails
Built-in rules query SecurityEvent; collect Windows security events from domain controllers	Windows Security Events via AMA with an event set or custom XPath	A generic Windows event DCR with the Security log: it writes Event
Full session auditing, including sign-outs, at reduced volume	The Common event set	Minimal: it omits sign-out event 4634
Collect System and Application logs	A generic Windows event DCR, which writes them to Event	All events: it covers only the Security and AppLocker channels
On-premises servers must send events through AMA	Install the Azure Arc Connected Machine agent first, then deploy the AMA-based connector	The Arc agent alone: AMA still does the collecting
Servers already forward to a Windows Event Collector	AMA on the collector with the Windows Forwarded Events connector, plus adapted rules or ASIM parsers	Expecting SecurityEvent rules to match unchanged: forwarded events land in WindowsEvent
A firewall can only send CEF over Syslog	CEF via AMA on a Linux log forwarder, with the installation script run	Installing AMA on the firewall: appliances cannot host the agent
A Syslog DCR set to LOG_ERR misses informational messages	Lower the facility's minimum log level	Recreating the DCR: the minimum level excludes less severe messages by design
Records appear in both CommonSecurityLog and Syslog	Separate the facilities, or add a transformation that drops CEF from the Syslog stream	Enabling every facility on both connectors, which causes the duplication
Every current and future resource of a type must stream logs	The connector's Azure Policy assignment plus a remediation task	Individual diagnostic settings: they do not cover resources added later
Collect subscription activity	The Azure Activity connector through diagnostic settings	Installing AMA on a VM: activity is a platform log
A partner exposes a TAXII 2.1 collection	Threat Intelligence - TAXII with the API root URL and collection ID	Only the discovery URL: it does not identify a collection
A TIP must push threat actors and relationships	The upload API with an Entra application that holds Microsoft Sentinel Contributor	The legacy Threat Intelligence Platform connector: it is indicator-only and being deprecated
Imported indicators raise no alerts	Enable a threat intelligence analytics rule whose required sources are connected	Re-importing the feed: ingestion is not detection
An application sends JSON that no built-in table fits	A _CL table, a Direct DCR in the workspace's region, and an app registration with Monitoring Metrics Publisher on the DCR	Sending to the workspace resource ID: the endpoint belongs to the DCR or a DCE
A CLI schema extension deleted existing custom columns	List every column to keep in --columns	Passing only the new column: the list replaces the set

When two options both sound plausible, ask which table the scenario's content queries and which stage the evidence proves; use those checks to compare the options.