Loading a subset of re-uploaded files can be a bit easier to manage with COPY INTO. See Schema inference and evolution for more details. If your data schema is going to evolve frequently, Auto Loader provides better primitives around schema inference and evolution. Auto Loader can discover files more cheaply compared to COPY INTO and can split the processing into multiple batches. If you are expecting files in the order of millions or more over time, use Auto Loader. If you’re going to ingest files in the order of thousands, you can use COPY INTO. Here are a few things to consider when choosing between Auto Loader and COPY INTO: The COPY INTO command is another convenient way to load data incrementally into a Delta table with exactly-once guarantees. When to use COPY INTO and when to use Auto Loader You don’t need to maintain or manage any state yourself to achieve fault tolerance or exactly-once semantics. In case of failures, Auto Loader can resume from where it left off by information stored in the checkpoint location and continue to provide exactly-once guarantees when writing data into Delta Lake. In fact, this is how Auto Loader can both perform a backfill on a directory containing existing files and concurrently process new files that are being discovered through file notifications. You can switch file discovery modes across stream restarts and still obtain exactly-once data processing guarantees.

This key-value store ensures that data is processed exactly once. The availability for these modes are listed below.Īs files are discovered, their metadata is persisted in a scalable key-value store (RocksDB) in the checkpoint location of your Auto Loader pipeline. See Leveraging file notifications for more details. File notification mode is more performant and scalable for large input directories or a high volume of files but requires additional cloud permissions for set up. See Incremental Listing for more details.įile notification: Auto Loader can automatically set up a notification service and queue service that subscribe to file events from the input directory.

In Databricks Runtime 9.1 and above, Auto Loader can automatically detect whether files are arriving with lexical ordering to your cloud storage and significantly reduce the amount of API calls it needs to make to detect new files. Directory listing mode allows you to quickly start Auto Loader streams without any permission configurations other than access to your data on cloud storage.

Optimized Amazon S3 Source with Amazon SQSĪuto Loader supports two modes for detecting new files: directory listing and file notification.ĭirectory listing: Auto Loader identifies new files by listing the input directory.

Scaling Auto Loader to large volumes of data.

Benefits over Apache Spark FileStreamSource.

When to use COPY INTO and when to use Auto Loader.Best practices: Delta Lake Structured Streaming with Amazon Kinesis.

Aws s3 copy wildcard