Aditi Kumari

Incremental ingestion of data is an ETL ideology none of us can rule out while building an ETL flow. In past, we have done this by maintaining a custom log of how many files have been read and where to start from next and the story becomes more and more interesting if you have to reprocess the data. All this can be summed up to a high latency, higher devOps cost and a burnt-out Data engineer. Autoloader simplifies this task and overcomes these limitations as an optimized file source by maintaining a file notification service at the back end. It can be run as a stream or used as a batch like process while all along feeding only the new data without getting on your nerves of how to maintain the state of processed files. In this session we will go through various ways of ingesting data incrementally and how far we have come to which in turn leads to the use of databricks component autoloader. We will peep in to the new features being added to this tool and how we can make use of it to our advantages.