The challenge
- Each REST source had different auth, pagination, rate limits, endpoints, and response formats.
- Some APIs required ID lookups before detail calls; others returned nested JSON.
- Needed one framework instead of one-off pipelines per source.
Our approach
- Built parent-child orchestration with ForEach loops for object-level extraction.
- Standardized pagination patterns: page, offset, continuation token, next-link.
- Landed raw JSON in ADLS by source, object, and run date.
- Added error handling and partial rerun design.
Results
1 framework
Reused across multiple APIs
Reduced
Development time per new source
Raw-first
Lake design supports multiple reports
Could this be your team next?
Tell us about your environment and we'll come back with a one-page diagnostic in 48 hours.
Get a free assessment
