Collect and store data

Suppose, for example, a churn-prediction team gathers customer rows from Amazon RDS, order history from Amazon DynamoDB, call recordings from an Amazon EFS share and click events from a stream; the DynamoDB table alone can be read with a Scan, which consumes read capacity units^[3], or with an export to S3, which does not^[4]. The AWS MLA-C02 exam guide^[5] names this work 'Task 1.1: Collect and store data.' (Data Preparation for ML and AI, 28%), with skills covering extraction, storage decisions and data formats; 'Task 1.2: Perform data transformation, feature engineering, and pre-processing.' (see Transform data and engineer features) and 'Task 1.3: Validate data quality and manage bias.' (see Validate data quality and manage bias) follow it.

Data sources and copy paths into Amazon S3

By how they are read, the sources form families (an organizing model, not an AWS classification):

• File storage includes Amazon EFS and NFS or SMB file servers. Amazon EBS sits outside this family: an attached EBS volume is exposed as a block device^[6] (a raw disk), and its files are read through the file system mounted on an EC2 instance.
• Databases and search indexes (Amazon RDS, Amazon DynamoDB, Amazon OpenSearch Service) are read by AWS Glue; an Amazon RDS read over a JDBC connection or an OpenSearch Service read runs against the live source, so size and schedule it with the source application's capacity in mind.
• Event streams (Amazon Kinesis Data Streams; Apache Kafka topics, run on AWS as Amazon Managed Streaming for Apache Kafka, or Amazon MSK) deliver records continuously. A streaming service captures them, and Amazon Data Firehose (formerly Kinesis Data Firehose) delivers them into storage.

AWS DataSync transfers NFS, SMB and Amazon EFS data to Amazon S3^[7]; AWS Glue for Spark reads Amazon RDS through a JDBC connection^[8], DynamoDB through its export connector^[9] and OpenSearch Service through an OpenSearch Service connection^[10]; Amazon Data Firehose reads a Kinesis data stream^[11] or an Amazon MSK topic and loads it into an S3 destination^[12].

Where the data lands

Amazon S3 is the landing place: first a raw copy of each source, then an analytical copy, meaning the same data rewritten in a columnar storage format such as Apache Parquet or Apache ORC^[13] and partitioned for the reads that feature jobs repeat. AWS Glue writes Parquet or ORC files to Amazon S3, partitioned while writing^[14], turning the raw copy into the analytical copy, as the figure below shows. Amazon S3 is also a source: a dataset already there can be read in place, for example by an AWS Glue job transforming data from Amazon S3 to Amazon S3, which needs no additional VPC configuration if it does not need to run in a VPC subnet^[8]. Two specialized stores sit downstream when the workload needs them: Amazon SageMaker AI Feature Store, which serves feature values to training and inference, and a vector store, which holds embeddings for retrieval in generative AI applications.

Takeaway, as a working heuristic rather than an AWS rule: identify how each source is read before choosing its copy path, then choose the storage layout by how the data will be read.

This section covers sources that hold files: Amazon EBS volumes, Amazon EFS file systems, and file servers that speak NFS or SMB. The rule is to move files as files and leave parsing to a later job.

Amazon EBS: read the mounted file system

An EBS volume is not an object store. After you attach an Amazon EBS volume to your instance it is exposed as a block device^[6] (a raw disk), and its files become readable only once the file system on it is mounted on the EC2 instance. The documented procedure formats and then mounts a volume, but formatting applies only to an empty volume. A volume created from a snapshot may already carry a file system and the dataset: check for an existing file system, then mount it without formatting and point the extraction process at the mounted path. Formatting it would overwrite the data you came to extract.

AWS DataSync: file servers and Amazon EFS

AWS DataSync copies files from a source location to a destination location. Whether a transfer needs a DataSync agent (a virtual machine you deploy next to self-managed storage) depends on the location pair: NFS or SMB to Amazon S3 requires an agent, while Amazon EFS or Amazon FSx to Amazon S3 does not^[7] for transfers within the same AWS account, and between AWS storage services in the same account no agent is required, whether in the same AWS Region or across AWS Regions in the same partition^[15] (here an AWS partition, which is a group of AWS Regions^[16], unrelated to the S3 and stream partitions later on this page). For an EFS source, DataSync mounts your file system as a root user from your VPC using network interfaces^[17], so the EFS location's subnet must be in the same Availability Zone as at least one mount target, and its security groups must allow inbound NFS traffic on port 2049. Treating managed EFS like an on-premises NFS server and deploying an agent adds infrastructure the transfer does not need.

DataSync moves bytes. It does not parse CSV rows, extract text or compute features; a downstream processing job does that once the files are in S3. The reverse mistake also appears in scenarios: an AWS Glue JDBC connection cannot copy an SMB document share, because JDBC connects to database records, not to file shares (the next section covers JDBC sources).

In short: an EBS volume is mounted, file shares and EFS move with DataSync, and the agent question is answered by the location pair, not by the protocol name alone.

Databases serve an application, so the extraction rule is: read the data without starving that application, and without re-reading what you already have. This section walks three database and index sources (Amazon RDS, Amazon DynamoDB and Amazon OpenSearch Service), each with a different mechanism.

Amazon RDS through an AWS Glue JDBC connection

An AWS Glue job reads a relational table through a JDBC connection that matches the database engine. Credentials are only half of the connection. When a job reads from a JDBC data store in a private subnet, AWS Glue sets up elastic network interfaces^[8] in the subnet named in the connection and applies the connection's security groups to them, and all JDBC data stores the job accesses must be reachable from that subnet. A connection timeout therefore calls for checking network configuration and endpoint reachability, including routing and security-group rules; rotating the database password changes neither.

For repeated runs, job bookmarks let Glue read only the rows added since the previous run by comparing saved progress with the values of one or more bookmark keys. User-defined bookmark keys must each be strictly monotonically increasing or decreasing^[18]; a revision number or minor-version field fails this test because its values repeat across the dataset. The rule has a consequence: an update to an old row whose bookmark key does not change is invisible to the bookmark, so a bookmarked incremental read using those unchanged keys misses the updates.

When the target is a data lake and nobody should have to write that job, an AWS Lake Formation blueprint generates it. Workflows generate AWS Glue crawlers, jobs, and triggers^[19] from a blueprint: the database snapshot blueprint loads or reloads all tables from a JDBC source, and the incremental database blueprint loads only new rows based on bookmark columns (Lake Formation's term for the columns that, like the bookmark keys above, track what was already loaded), which the same page recommends only when previous rows are not updated. A Kinesis data stream is not a substitute for either: producers put records into Amazon Kinesis Data Streams^[1], so something would still have to read and publish every database row.

Amazon DynamoDB: export instead of scan

A table scan reads through the table's capacity: Scan operations consume read capacity units^[3], and a parallel scan can consume available read capacity faster. The export path avoids this. DynamoDB export to S3^[4] requires point-in-time recovery (PITR) on the table, writes the table as of any time in the PITR window to an S3 bucket, runs asynchronously, and does not consume read capacity units. Export and S3 storage charges still apply, and an asynchronous export completes when it completes, not at a guaranteed time.

The AWS Glue DynamoDB export connector^[9] requests that export on behalf of a job, and the exported data persists in S3 after the job run finishes, so later jobs can reuse it without additional DynamoDB exports. Reuse the export when several preprocessing experiments should see the same snapshot; request a new export when the dataset must reflect later table changes.

Amazon OpenSearch Service through a Glue connection

When documents already indexed in OpenSearch must feed a dataset, AWS Glue for Spark can read from OpenSearch Service in AWS Glue 4.0 and later versions^[10]. You name the source index, set pushdown, which instructs Spark to pass read queries down to OpenSearch so the database only returns relevant documents^[10], and can supply an OpenSearch query in opensearch.query so that only matching documents come back as a DynamicFrame (Glue's distributed table structure). The same page states that this feature is not compatible with OpenSearch Serverless, so confirm the source is an OpenSearch Service domain before choosing it.

Takeaway: of these mechanisms, only the DynamoDB export reads no table capacity; a JDBC read, bookmarked or not, still queries the live database, and a pushed-down OpenSearch query still runs on the domain, so size and schedule those two with the application's load in mind.

When two sources in S3 need to become one table, two common ways to combine tabular datasets are side by side or end to end. AWS Glue's documentation puts it in one line: you use Union to combine rows, and you use Join to combine columns^[20].

• A join matches rows from two datasets on a key and places their columns side by side: each transaction gains the attributes of the customer it belongs to.
• A union appends rows that share one logical schema: this month's transactions stacked under last month's. Align column meanings and data types before appending. Appending customer profiles under transactions attaches no customer to any transaction; it only mixes two kinds of row in one table.

Which join keeps which rows

The join type decides which training examples survive. In SQL terms, an inner join returns matching rows only, and an outer join returns those rows plus the non-matching rows from the left table, the right table, or both, with NULL values filling the gaps^[21].

Join type	Rows in the output	Use it when
Inner	Only keys present in both datasets	Every example must have the enrichment
Left (outer)	Every left row, with NULL right-side columns where unmatched	Every training example must survive enrichment
Cross	Every left row paired with every right row	You deliberately want all combinations, such as a candidate grid

Put the training examples on the left of a left join when optional profile fields are being added; an inner join would silently drop every example whose profile key has no match. A left join cleans nothing, though: inspect the NULL enrichment values, and check the right side for repeated keys, which duplicate a left row once per match. A cross join is the trap at the other extreme: cross-joins are unqualified joins that return the Cartesian product of the two tables^[21], so using one to attach customer attributes to transactions multiplies rows instead of matching them, and inflates intermediate data and processing cost.

Running the merge in AWS Glue

AWS Glue jobs authored in AWS Glue Studio run on the Apache Spark-based serverless ETL engine in AWS Glue^[22], so a merge scales out without a cluster to manage. In the visual editor, the Join transform combines two datasets into one on the key names you specify^[23]; it accepts exactly two parent nodes and offers inner, left, right, outer, left semi and left anti joins. The Union transform requires all inputs to have the same schema^[20] (an ApplyMapping transform can align mismatched schemas first) and has two types: ALL, the default, keeps duplicate rows, and DISTINCT removes them. Appending sources does not deduplicate by itself; choose DISTINCT when the merged output must not repeat rows.

Takeaway: decide rows versus columns first (union or join), then decide which unmatched rows must survive (the join type), and only then pick the tool that runs it.

This section covers the two layout decisions for data at rest in S3: the file format and the partition scheme. Both follow one rule: choose by how the data will be read, not by how it arrived.

Formats by access pattern

Format	Layout	Choose it when
Apache Parquet, Apache ORC	Columnar, compressed per column	Jobs repeatedly read a few columns of a wide table
JSON	Named fields, nested objects and arrays	Records are semi-structured or nested
CSV	Flat rows and columns	Simple interchange with a consumer that expects it

Parquet and ORC store each column's values together. That brings compression by column, with the compression algorithm selected for the column data type^[13], and predicate pushdown, which lets a query engine such as Amazon Athena fetch only the blocks it needs. A feature job that reads a handful of columns from a wide table can therefore read less data from a columnar copy than from CSV, where every row must be read in full. CSV stays useful for interchange; being human-readable does not make it efficient for repeated wide-table scans.

JSON keeps nested structure that CSV cannot express without an agreed flattening or encoding convention. The deciding question is what the consumer accepts. SageMaker AI built-in algorithms, for example, take content types such as text/csv, application/json and application/x-recordio-protobuf, and XGBoost supports only text/csv from that list^[24] (plus text/libsvm). Nested JSON is no use to an algorithm that reads only CSV.

Converting a stream to Parquet on the way in

Amazon Data Firehose can write the columnar copy directly: it can convert the format of your input data from JSON to Apache Parquet or Apache ORC before storing the data in Amazon S3^[25], using a schema it reads from the AWS Glue Data Catalog. Two constraints follow from the same page. The input must be JSON, so CSV records are first transformed to JSON with an AWS Lambda function. And the catalog schema governs the output: attributes that are not in the schema do not appear in the converted data.

Partitions that match the query filters

An S3 partition is a group of files under a shared key prefix such as year=2026/month=10/. (It is unrelated to a stream's partitions, such as the shards that make up a Kinesis data stream^[1].) By partitioning your data, you can restrict the amount of data scanned by each query^[26]: when a query names the partition in its WHERE clause, Athena scans only that partition. Partition by the fields downstream queries filter on, most often event date, sometimes a customer or region field. Partitions and columnar formats are complementary: partitions decide which files are read, and the columnar format decides which columns are read within those files.

Takeaway: an analytical copy is Parquet or ORC, partitioned by the query filters; JSON and CSV serve the consumers that need them.

The AWS MLA-C02 exam guide^[5] asks for storage decisions 'based on cost, performance, data structure, and data compliance'; for data in S3, the storage class settles the cost-versus-access part of that choice.

The storage class must match when the data will be read next: archive classes offer low storage rates for suitable rarely accessed data, but S3 Glacier Flexible Retrieval and S3 Glacier Deep Archive objects must be restored before a job can read them. The table lists the classes this page uses; S3 offers others, such as S3 One Zone-IA and S3 Express One Zone.

Storage class	Access	Fits a dataset that is
S3 Standard	Milliseconds	Read often
S3 Standard-IA	Milliseconds, with a retrieval fee	Read rarely but needed at once when read
S3 Intelligent-Tiering	Milliseconds in its default tiers	Read on a pattern nobody can predict
S3 Glacier Instant Retrieval	Milliseconds	Archived but occasionally needed at once
S3 Glacier Flexible Retrieval, S3 Glacier Deep Archive	Restore first	Archived and read only after a planned restore

S3 Glacier Flexible Retrieval and S3 Glacier Deep Archive objects are archived, and not available for real-time access^[27], while S3 Glacier Instant Retrieval serves long-term data that requires millisecond retrieval. Putting the only copy of next week's training data in Deep Archive without completing a restore before the run breaks the immediate-read requirement; the lower storage charge does not remove the need to plan that restore.

S3 Intelligent-Tiering, for a small monthly monitoring and automation fee, watches access patterns and automatically moves objects between access tiers as access changes; its optional Archive Access and Deep Archive Access tiers again require a restore before reading. S3 Standard-IA is a different kind of choice: a class you select because you already know the access is infrequent, not a mechanism that moves anything automatically.

Takeaway: restore-based archive classes trade access time for lower storage rates, so choose the class from the date of the next read.

Suppose, for example, a training-set version must stay unaltered for a fixed period no administrator can shorten, and an open-ended investigation may extend that; S3 Object Lock^[28] covers both, blocking deletion and overwrite of object versions, and works only in buckets with S3 Versioning enabled. It has two retention modes and legal holds:

• Compliance mode: a protected object version can't be overwritten or deleted by any user, including the root user in your AWS account^[28], and its retention period can't be shortened. Use it when nobody, administrators included, may bypass retention.
• Governance mode: users cannot overwrite or delete a locked version unless they have special permissions; a user with s3:BypassGovernanceRetention who sends the bypass header can. It protects against most users, not against authorized administrators.
• Legal hold: the same protection as a retention period, but with no expiration date^[28]; it remains until someone explicitly removes it. Use it for an investigation whose end date is unknown, where a fixed retain-until date could expire too early.

Legal holds are independent of retention periods, so when a retention period expires an active legal hold still protects the version. Object Lock also protects versions, not keys: retention periods and legal holds don't prevent new versions of the object from being created. A PUT to the same key creates a new version while the locked version stays intact.

Takeaway: compliance mode is the only Object Lock retention no one can bypass, and a legal hold protects a version until someone removes it.

A model can be reproduced only if the exact data it was trained on can be found again. The AWS Well-Architected Machine Learning Lens calls for automated version control for training data, so you can precisely recreate a model version and keep a clear audit trail of data transformations^[29]. In practice, record for each prepared dataset version its source identities, the source versions or extraction points (an export time, a bookmark position), and the transformations applied, and retain the relevant source and output versions under the dataset's retention policy. Source metadata says where data came from; transformation lineage says how it became the prepared artifact. Keeping only the preparation code while overwriting inputs and outputs preserves the logic but loses the data an earlier run used.

Takeaway: reproducibility needs versioned data plus its lineage, not just versioned code.

Suppose, for example, that two teams must read the same card-payment events independently; with Kinesis Data Streams, each application can consume data from the stream independently and concurrently^[1]. Amazon Kinesis Data Streams and Apache Kafka (run on AWS as Amazon Managed Streaming for Apache Kafka, or Amazon MSK) both serve that need, and they share one model, so learn the model once. Think of each as a partitioned log (a general streaming term, not an AWS feature name): the stream is split into partitions, each partition is an append-only sequence of records. Kinesis calls the partitions shards; Kafka calls them topic partitions. A stream partition has nothing to do with an S3 partition, the file-prefix grouping from Choosing file formats and partition layout above.

Kinesis Data Streams: shards and partition keys

A Kinesis data stream is a set of shards^[1], and Kinesis places records on shards in one of two ways. In partition-key-based distribution, the producer attaches a partition key to each record and that key picks the shard. In service-managed distribution (the AUTO setting), a partition key a producer still supplies is one the service accepts and ignores for shard placement^[30], so no ordering relationship between records may be assumed^[31]. Everything below about keys, order and hot keys assumes partition-key-based distribution.

Kinesis maps the key through an MD5 hash to choose the shard, unless an explicit hash key overrides that mapping. Within the shard each record receives a sequence number that is unique per partition key within its shard, and the key concepts page^[1] notes that sequence numbers cannot be used as indexes to sets of data within the same stream. Sequence order is shard-local, not a guarantee of source-event order across the stream. Keep events whose relative order matters under one partition key, account for producer retries that can reorder writes, and never sort a multi-shard stream by sequence number as though it were one global counter.

Each shard provides sustained capacity of up to 1,000 records or 1 MB per second for writes, and up to 2 MB per second for reads^[1]; intermittent large records can temporarily burst beyond the sustained write throughput^[32]. Write-side and read-side capacity problems have different fixes:

• Hot partition key (producer side). One sufficiently hot key can exceed a shard's write capacity while other shards have spare capacity. The producer troubleshooting guidance^[33] is to choose a partition key that distributes puts evenly across all shards and to check whether one key is hitting the limits while the rest are not. Adding shards does not fix a single key exceeding one shard's write capacity; spread that traffic over more key values.
• Reader contention (consumer side). Default shared-throughput consumers split each shard's 2 MB per second read allowance. Enhanced fan-out^[2] gives each registered consumer up to 2 MB per second per shard of dedicated throughput, with up to 20 registered consumers per stream in On-demand Standard and Provisioned modes (50 in On-demand Advantage mode). It fixes readers competing with each other, not a hot key on the write side.

Records stay readable for the stream's retention period, 24 hours by default and up to 8,760 hours (365 days)^[1], and each consumer tracks its own position. That makes Kinesis Data Streams a replay log: several applications can read and re-read the same records within retention. Amazon Data Firehose, by contrast, delivers records to destinations and offers no such consumer model.

Kafka: topics, consumer groups and offsets

Kafka maps onto the same model. Producers append records to a named topic divided into partitions, and consumers read each partition sequentially. Amazon EventBridge Pipes, an AWS service that connects sources to targets^[34], is one Kafka consumer among many: when it consumes from Amazon MSK, it reads the messages sequentially for each partition^[35]. Ordering, again, is per partition; parallel partitions do not form one ordered topic log.

A consumer group is the set of workers sharing one workload. When other active consumers share a consumer group ID, Apache Kafka distributes messages across all consumers^[35], rather than broadcasting every message to every member, and to preserve per-partition order the maximum is one consumer per partition. Two rules follow: the partition count caps useful parallelism within a group, and independent applications that each need the whole stream use separate consumer groups. Raising the topic's replication factor adds copies of each partition for durability (Amazon MSK recommends a replication factor of at least 3^[36]); it adds no partitions for the group to share.

A committed offset is the group's saved position in each partition. A consumer that rejoins an existing group ID begins processing records according to the committed offset of the consumer group^[35], so keep the group identity when restarting an ingestion consumer. A saved offset does not keep records alive: replay works only while the records are still within the topic's retention.

The figure below shows Kinesis partition-key-based distribution: partition keys hash to shards, and consumers read either through the shared allowance or through dedicated enhanced fan-out throughput.

Takeaway: for throttling or ordering problems, start with the relevant shard or partition; a few limits, such as the number of registered enhanced fan-out consumers, apply per stream.

A streaming feature such as 'payments per card in the last 5 minutes' raises a time-window question: which clock defines the window? This section covers that choice in Amazon Managed Service for Apache Flink^[37] (formerly Amazon Kinesis Data Analytics for Apache Flink), which runs Apache Flink applications over streams such as Kinesis Data Streams or Kafka topics. Stream transforms are a separate skill in the AWS MLA-C02 exam guide^[5], 'Skill 1.2.3: Transform streaming data (for example, by using AWS Lambda, Spark).' Flink distinguishes three notions of time^[38]:

Time basis	Meaning	A window then reflects
Event time	When each event occurred on its producing device	What happened when
Ingestion time	When the event entered the Managed Service for Apache Flink service	When data arrived
Processing time	The system time of the machine executing the operation	When the operator ran

A payment delayed by the network belongs to the minute in which it happened. With processing-time windows a delay that crosses a window boundary can place it in a later window; with event-time windows it can be counted where it occurred, subject to the watermark (which propagates information about event time from the source to all other operators^[39]) and to how the application handles late events.

Takeaway: choose the time basis that matches the feature's business meaning, which for statistics about when events occurred is event time.

Amazon Data Firehose is the managed path from a stream into storage: it delivers real-time streaming data to destinations such as Amazon S3, Amazon Redshift and Amazon OpenSearch Service^[40] with no applications or resources to manage. Its delivery model is the opposite of the replay log in the Kinesis and Kafka section: Firehose pushes records into a destination; it does not keep a log for consumers to re-read.

Buffering decides freshness

Firehose collects records before writing them. It buffers incoming streaming data in memory to a certain size and for a certain period of time before delivering it^[41], and whichever buffer condition is met first triggers delivery. Larger buffers produce fewer, larger objects, which suit downstream readers. For a low-volume stream, though, the time condition is the one that fires, so a longer interval only delays when records appear in S3. Firehose accepting a record is not the record being visible at the destination; set the interval from the freshness the consumer needs. When dynamic partitioning (building S3 prefixes from record fields, described in the next subsection) is enabled, multi-stage buffering, which Firehose uses internally to deliver maximum size objects^[42], means the end-to-end delay might be 1.5 times the configured buffer interval.

Dynamic partitioning: prefixes from record content

Dynamic partitioning builds each record's S3 prefix from fields inside the record, so delivered objects are already laid out as S3 partitions. Partition keys come either from inline parsing with a jq expression for JSON records, or from an AWS Lambda function^[43], and the S3 prefix expression references them, for example customer_id=!{partitionKeyFromQuery:customer_id}/. In that example, customer_id= is literal prefix text, and the expression format is !{namespace:value}, where the namespace is partitionKeyFromQuery for keys created by inline parsing or partitionKeyFromLambda for keys returned by a Lambda function^[44]; the value, customer_id again, is the key's ID. Firehose evaluates the expression at runtime and groups records that match the same evaluated S3 prefix expression into a single data set^[44]. Deriving year, month and day from the record's own event timestamp files a late event under the day it occurred, not the day it arrived: the event-time idea from the previous section.

Every distinct evaluated prefix is an active partition with its own buffer. A new buffer is created for each active partition^[42], each buffer is delivered as its own object when its own size or time condition is met, and a partition Firehose cannot deliver stays active until it can. Once the number of active partitions reaches a limit of 500 per Firehose stream (the API still calls the resource a delivery stream: CreateDeliveryStream creates a Firehose stream^[45]), the remaining records go to the S3 error bucket prefix (error type activePartitionExceeded); the quota can be raised to 2,500 on request. The capacity trap here is cardinality, not throughput: partitioning modest traffic by a high-cardinality field such as customer ID creates many small buffers, many small objects and a high active-partition count. Lengthening the buffer interval makes this worse, because each buffer stays active longer.

The S3 error prefix

Enabling dynamic partitioning requires an S3 error bucket prefix. If Firehose cannot parse a record, extract the specified partitioning keys, or evaluate the expressions in the S3 prefix, the record is delivered to the S3 error bucket prefix^[46], organized by error type and delivered together with information about the error. A producer that renames or drops the field a key is extracted from shows up here: compare the failed records' field names and structure with the extraction expression, then fix the producer or the expression. Requesting a higher active-partition quota does nothing for a missing-key error.

Processing order, record boundaries and Redshift

KPL is the Kinesis Producer Library, whose built-in aggregation is available when the Firehose stream has a Kinesis data stream as its source^[47]. Firehose applies its optional stages in a fixed order before the final Amazon S3 delivery: KPL (protobuf) deaggregation, JSON or delimiter deaggregation, Lambda processing, data partitioning, data format conversion, and Amazon S3 delivery^[47]. The figure below traces that order and the error-prefix branch. Two settings touch record boundaries, in opposite directions:

• Multi-record deaggregation splits input. Several events aggregated inside one record payload sent with PutRecord or PutRecordBatch must first be deaggregated^[48] before dynamic partitioning can route each one.
• New line delimiter separates output. It adds a new line delimiter between records in objects that are delivered to Amazon S3^[47], which readers such as Athena need to tell aggregated records apart. Parquet conversion, from Choosing file formats and partition layout, produces a columnar format instead and does not meet a newline-delimited text requirement.

For an Amazon Redshift destination, streaming data is delivered to your S3 bucket first, and Firehose then issues an Amazon Redshift COPY command^[40] to load it. The S3 bucket is a required staging step, and a successful S3 delivery does not yet mean the Redshift load succeeded. The optional source-record backup, offered when data transformation is enabled, is a separate copy with a separate purpose.

Takeaway: Firehose trades latency for batch size through its buffers, and its distinctive failures, active-partition exhaustion and key-extraction errors, both land in the S3 error prefix, where the error type tells you which one you have.

A retrieval-augmented generation (RAG) application turns documents into embeddings (lists of numbers that place similar content close together) and, at query time, finds the stored vectors nearest to the query's vector. The AWS MLA-C02 exam guide lists producing embeddings as 'Skill 1.2.5: Configure and use embedding models to transform text and image data into numerical representations.' (see Transform data and engineer features) and storing them as 'Skill 1.1.7: Configure scalable vector databases for AI applications (for example, OpenSearch Service, Amazon RDS with pgvector, Amazon S3) based on specifications.' The three AWS options below illustrate how different requirements guide the choice:

• Amazon OpenSearch Service, when keyword search and vector similarity are both required. OpenSearch supports lexical search, vector search, and hybrid search that blends traditional lexical search with vector-based semantic search^[49].
• Amazon RDS for PostgreSQL with pgvector, when an existing PostgreSQL application must keep vectors next to its relational data and query both with SQL. pgvector is an open-source extension that adds the ability to store and search ML-generated vector embeddings^[50]; use an RDS for PostgreSQL version that supports it.
• Amazon S3 Vectors, when the requirement is S3-native vector storage: purpose-built, cost-optimized vector storage with a dedicated set of API operations and no infrastructure to provision^[51].

Moving relational records into an OpenSearch index only to gain vector search breaks a stated requirement to keep SQL integration inside the existing database; replacing exact keyword matching with similarity search breaks a requirement for both behaviours. One related distinction: if the vectors must back an Amazon Bedrock knowledge base, the PostgreSQL option in Bedrock's vector store setup documentation^[52] is Amazon Aurora PostgreSQL, alongside OpenSearch and S3 Vectors among others.

OpenSearch Service: k-NN indexes

In OpenSearch Service, k-NN, short for its associated k-nearest neighbors algorithm, finds the nearest neighbors for points in a vector space^[53]. You create an index with the index.knn setting and add one or more fields of the knn_vector data type^[53], each with a required dimension, and search by Euclidean distance or cosine similarity. Keep the searchable text and structured fields in the same documents so that lexical queries and filters work alongside similarity. Embedding generation remains a separate preparation step unless you configure a model integration; OpenSearch integrates with Amazon Bedrock and Amazon SageMaker AI for generating embeddings^[54].

Comparing a query with every stored vector (exact k-NN search) gets slower as the collection grows. Approximate k-NN uses algorithms like HNSW (Hierarchical Navigable Small World) for faster searches on large datasets^[54], at the price of possibly missing a neighbour that exact search would return; OpenSearch documentation describes its tuning as a balance of recall versus latency^[55]. Measure recall as well as latency, keep exact search for cases where the result guarantee and dataset size justify exhaustive comparison, and do not treat approximate results as identical to exhaustive search just because both return k neighbours.

S3 Vectors: buckets, indexes and fixed settings

S3 Vectors introduces the vector bucket, a bucket type purpose-built to store and query vectors, and vector indexes inside it where similarity queries run^[51]. Uploading embedding arrays as ordinary objects into a general purpose bucket stores the bytes but creates no similarity index: listing objects is not nearest-neighbour search.

Besides the index name, three settings are chosen at creation, and after you create a vector index, you can't update the vector index name, dimension, distance metric, or non-filterable metadata keys^[56]; changing one means creating a new index.

• Dimension: the number of values in a vector, an integer between 1 and 4,096. All vectors added to the index must have exactly this number of values, so set it to the embedding model's output dimension. A model with a different output dimension needs a new index; padding shorter vectors with arbitrary values changes what they represent.
• Distance metric: Cosine measures the angle between vectors and suits normalized vectors where direction matters more than magnitude; Euclidean measures straight-line distance and suits cases where magnitude matters too. Follow the embedding model's recommendation.
• Non-filterable metadata keys: metadata is filterable by default. Keys you declare non-filterable are returned with results but cannot be used as query filters; the original source text is the documented example of a good candidate. Never mark a field you must filter on, such as document category, as non-filterable.

Takeaway: choose the store by the queries that must run beside similarity search, then match dimension and metric to the embedding model and choose filterable metadata for the application before loading anything.

Machine learning and generative AI workloads ingest more than tables. The AWS MLA-C02 exam guide^[5] lists 'Skill 1.1.8: Ingest and store diverse data types (for example, text, images, audio) for AI and ML applications.'

Text: documents, web pages and knowledge bases

A PDF can have an embedded text layer that ordinary PDF text extraction reads, including text added by optical character recognition (OCR) to scanned pages, or pages that are only images. A text-only parser finds nothing to extract from an image-only scanned page without an existing text layer; OCR has to run first. Amazon Textract analyzes image files and PDF files^[57] and returns the lines and words of detected text, the page they appear on, and their location on the page^[58]. Inspect the output, because document layout can affect reading order and how fields are interpreted.

Web pages can carry boilerplate and navigation menus. Extract the passages that bear on the page's content rather than keeping a raw HTML dump, so that a model sees the parts that matter and spends less of its input on boilerplate and navigation^[59], and keep the headings, because clear headings and subheadings help RAG models understand the structure of documents^[60]. When menus repeat across pages, a dump of every visible string can overrepresent those menus rather than each page's subject.

When documents feed an Amazon Bedrock knowledge base, editing the files in the data source is not enough: each time you add, modify, or remove files, you must sync the data source, and syncing is incremental^[61], so only documents added, modified or deleted since the last sync are processed. Recreating the knowledge base after every edit is unnecessary.

Images: decodable assets linked to their origin

An image corpus is usable only if its files decode, in formats and dimensions the consuming workflow supports. SageMaker Data Wrangler^[62], now used through Amazon SageMaker Canvas^[63], imports images from an S3 directory and offers built-in transforms including ResizeImage, DropCorruptedImages and DropImageDuplicates. The dataset-versioning rule^[29] from Dataset versions and lineage above applies to images too: keep the originals instead of overwriting them with resized copies, and keep a reference from each prepared image to its source. An Amazon SageMaker Ground Truth input manifest records such references: each JSON line's source-ref key names the Amazon S3 object, used when the object is binary, such as an image^[64]. Labeling is its own exam-guide skill, 'Skill 1.3.2: Label and annotate data.' (see Validate data quality and manage bias). Replacing images with OCR text keeps the characters and loses the visual content an image model needs.

Audio: Amazon Transcribe input and output

Amazon Transcribe^[65] has two modes with different rules: batch transcription reads media files stored in an S3 bucket, and streaming transcription reads a live media stream. Their input requirements differ:

Requirement	Batch	Streaming
Supported formats	AMR, FLAC, M4A, MP3, MP4, Ogg, WebM, WAV	FLAC, Ogg Opus, PCM encoding
Sample rate in the request	Optional	Required

A WAV file that a batch job accepts is not a valid stream as it is: the same page recommends PCM signed 16-bit little-endian audio for streaming and notes that this does not include WAV. Whenever a sample rate is supplied, it must match the actual audio; a mismatched value can make the job fail, and raising the declared number does not resample a low-fidelity recording.

Two features separate speakers, for two different recordings. Speaker partitioning (diarization) labels multiple speakers who share one audio channel; channel identification handles speech that is already on separate channels. Choose by how the audio is laid out, not by how many transcript files you expect.

For transcripts that become training data, choose the output location deliberately. If you name an S3 bucket you own and give Transcribe write permission, the transcript remains in that bucket until you remove it. Without one, Transcribe uses a service-managed bucket and returns a temporary URI that is valid for 15 minutes, and the transcript is deleted when the job expires after 90 days. A temporary download link is not a durable dataset location.

Takeaway: choose the steps each input needs beyond file copying (OCR or boilerplate removal for text, decoding and resizing for images, mode and encoding checks for audio), and for this training pipeline, the durable copy should go to storage you control.

Amazon SageMaker AI Feature Store keeps feature values so that training and inference read the same data. The AWS MLA-C02 exam guide lists 'Skill 1.1.9: Ingest data into SageMaker Feature Store.' separately from 'Skill 1.2.2: Create and manage features by using AWS tools (for example, SageMaker Feature Store).'; for the second, see Transform data and engineer features.

Two stores, two read patterns

A feature group, Feature Store's main resource, can be configured with an online store, an offline store, or both. The online store retains only the latest records and is designed for real-time predictions that need low millisecond latency reads; the offline store keeps all records as a historical database for data exploration, model training and batch predictions^[66], and it stores that history in your S3 bucket. Querying the offline store for every latency-sensitive inference request substitutes an analytical scan for the lookup the online store exists to serve.

The feature group contract

Before anything is ingested, the feature group defines its record schema. Each feature definition consists of a name and one of the data types integral, string or fractional^[67]. The record identifier names the feature that identifies the entity, such as a customer ID. The event time is the timestamp of when the record's event occurred, and every record must carry one. A combination of record identifier and event time uniquely identifies a record within the feature group. Validate incoming data against this contract; a feature group is not a schemaless document collection. Using an ingestion job's run ID as every row's record identifier collapses unrelated entities under one lookup key.

Streaming and batch ingestion

Feature Store accepts data in two ways: streaming, where records are pushed with the synchronous PutRecord API as soon as an update is detected, or batches^[66], for example features authored in SageMaker Data Wrangler and ingested by a SageMaker AI Processing job. Both populate the same feature groups, so a batch arrival pattern is no reason to build a separate feature repository.

Event time decides what is current

Event time here is the same idea as the Flink event time above: when the event happened, not when the record arrived. If a new record's EventTime is greater, the new record is written to both the OnlineStore and OfflineStore; otherwise, the record is a historic record and it is written only to the OfflineStore^[68]. The figure below traces that decision. A historical backfill therefore lands in the offline history without overwriting newer online values, and the last record to arrive does not become current merely by arriving last.

To restrict a write explicitly, the TargetStores request parameter names the store or stores to write; by default, Feature Store adds the record to all of the stores the feature group uses. Targeting only the offline store is the clean way to load history. Targeting cannot create a store the feature group was never configured with.

Whole-record writes

PutRecord writes a whole record: send every feature value the resulting record must contain, because omitted features are not merged in from the previous version. The Standard_V2 online store tier adds partial updates: it supports partial updates to individual features using the UpdateRecord operation, without rewriting the entire record^[69], which the Standard tier does not.

Reading history back for training

The offline store is an append-only store, and its data is stored in Parquet format^[66], so a raw query returns every version of every entity. An entity-level training snapshot needs one chosen version per entity at a chosen point in time. The SageMaker Python SDK's create_dataset() helper does not include deleted or duplicated records by default^[70] and offers selectors such as as_of, which builds the dataset up to a timestamp you specify. Using every historical row as a current independent training example mixes feature versions into the training population.

Takeaway: define the contract (record identifier, event time, data types), ingest by stream or batch into the stores you configured, and let event time, not arrival order, decide what is current.