FAQs on Azure Data Factory Config Driven Framework
FAQs on Azure Data Factory Config Driven Framework
1. What is config driven approach?
The configuration-driven approach moves the drivers of logic in our code from the code itself to a table reserved only for things we can configure. With config driven approach, you don’t need to modify the code every time to apply changes; you can plug in a new configuration to apply the changes.
2. What’s the need for a configuration-driven ELT framework?
- Development speed
Minimize effort to load & transform data
With loads of information available at your disposal, it’s easy to use and learn.
Building a new framework will help enforce the best practices and enterprise-level standards for better security and data governance policies.
- Identify changes quickly
With the framework in place, developers can have source control and quickly identify the changes in config files.
The framework is reusable; it’s not specified to one team; different teams can use the framework and scale on an enterprise level.”
3. Does ADFCDF handle ETL data migration?
Yes, it can handle ETL/ELT data migration with low code; AnblicksCDF was developed to address the present-day data integration challenges such as increased development time, operational inefficiencies, maintenance cost, and performance issues.
4. Is Azure Data Factory Config Driven Framework support all kinds of data sources?
The tool is config-driven instead of code-driven. It has built-in support for RDMS, APIs, flat files, and semi-structured data sources like JSON or XML.
5. Does ADFCDF supports cloud migration?
ADFCDF can migrate from OnPrem to Cloud, data stage, or legacy technology to Azure migration, version, and JDK upgrades.
6. Can ADFCDF be used for any database?
ADFCDF is a config-driven framework. It supports all relational databases and Files(XML, JSON, Flat Files) (Oracle, SQL). Cloud Related databases such as AWS, Oracle, Azure, etc.
7. How long does a typical migration take?
Migration time would depend on the data size and the server capacity. ADFCDF can run all the tasks parallelly, which helps reduce the migration time.
8. How long does the migration of the legacy ETL to new systems take with the ADFCDF framework?
Objects such as Database tables, Stored Procedures, Views, and Functions are configurable in the metadata table. It’s about time to add x number of objects into the metadata table.
9. Can I run both systems (the old and the new) until I’m happy to make the switch over?
Our certified Azure experts won’t be touching the existing target database. We point the ADFCDF to a new database that replicates the existing database, and customizations can be done based on your need. So there would be no impact on the existing systems, and both systems (old and new) can run parallelly. Once the replication is done without any issues, we can switch to the new system.
10. Why should I use ADFCDF instead of my self-managed replication solution?
- Cost-effective solution with reduced maintenance effort
- Add new data sources, pipelines with little configuration or no input
- Improved performance with configurable Task parallelization
- Insightful, proactive operational dashboard to mitigate failures
11. Can I monitor the progress of a database migration task?
With comprehensive logging and error handling, users can monitor the task details to ensure smooth BAU operations.
12. How can ADFCDF handle the errors??
Azure Data Factory Config Driven Framework is a comprehensive error handling function that will capture all the failed tasks in a log table with an error message. Once the error is resolved, the task will restart from any point of failure without manual intervention.
13. How does the migration team assist me with moving my data with ADFCDF?
We are Microsoft’s Gold Partner offering certified Azure professionals. Our best practices & reusable frameworks help you optimize Azure data factory development and execution configurations. We help you migrate from OnPrem to Cloud, data stage, or legacy technology to Azure migration, version, and JDK upgrades. Our deep Azure ETL design and big data expertise accelerate all your data and ETL/ELT initiatives with low maintenance and minimum costs.
14. Our company needs a custom program to automate something – can you develop it?
ADFCDF’s configurable workflow will allow users to build custom programs based on the requirements. For Eg: We helped a client build a custom program that will automate the conversion of XML/JSON files to CSV.
15. How does ADFCDF reduce ETL job development time?
Users can decrease their ETL development time halfway thanks to the metadata-driven framework and task reusability feature. However, when it comes to metadata-driven ETL development, there is no set formula to follow and no one-size-fits-all approach. This is why it’s critical to work with a development team like Anblicks that knows exactly what they’re doing.
16. Can I develop my operational dashboards?
We integrated the accelerator with PowerBI, a data visualization application to create interactive and out-of-the-box operational dashboards for efficient troubleshooting and error monitoring. With the help of the log table, users can create their operational dashboards.
17. Can I develop my Operational dashboards?
Yes, Users can integrate ADFCDF with other data visualization applications like Kibana, Tableau, Looker, etc.
18. Are the metrics shown in the operational dashboard real-time?
With our comprehensive logging and error handling mechanism, users can see the metrics in real-time.
19. Can I add other programming scripts?
Yes, you can add java, python, Unix, etc.
20. How Anblicks Config Driven Framework can help accelerate development and reduce maintenance?
As a Microsoft gold partner, we aim at providing customers with the ability to solve disparate data integration challenges and transform their business operations with data-driven decision-making. Our certified Azure architects bring deep Azure design and big data expertise to help you accelerate all your data and ETL/ELT initiatives.
Anblicks config-driven framework is a Low Code Enterprise ETL/ELT Framework powered by Azure ETL data factory. Its comprehensive logging and error handling mechanism allows parallel task execution to reduce development time. The out of the box operational dashboard helps in efficient troubleshooting and monitoring of failed tasks.
Please refer to the below picture to learn more about the features and benefits of Azure Data Factory CDF.
Are you interested in learning more about ADFCDF or how it can help your organization? Schedule a Free Demo
Saikiran Bellamkonda is a digital marketing ninja,works as a Sr Digital Marketing Executive at Anblicks. He focuses on developing, implementing, & managing innovative Inbound and Outbound marketing initiatives. He also applies his deep knowledge and experience to write about Data integration, warehouse, analytics, and visualization. He is a gym rat and a travel enthusiast.