Luigi is a data pipeline library written in python for batch processing jobs handling dependency management, workflow management, visualization and failures using few lines of commands. Luigi was developed by Spotify, later they open sourced it and now is used by variety of companies to automate their day to day tasks. The purpose of Luigi is to make things easy by providing more flexibility and ease unlike the cloud platforms that provide data pipelines as service, those could be very easy to use but with less features and functionality plus lack of portability.
Luigi data pipelines for batch processing
Luigi data pipelines for batch processing
Luigi data pipelines for batch processing
Luigi is a data pipeline library written in python for batch processing jobs handling dependency management, workflow management, visualization and failures using few lines of commands. Luigi was developed by Spotify, later they open sourced it and now is used by variety of companies to automate their day to day tasks. The purpose of Luigi is to make things easy by providing more flexibility and ease unlike the cloud platforms that provide data pipelines as service, those could be very easy to use but with less features and functionality plus lack of portability.