This is a great discussion. Here’s one more aspect that may be helpful as you’re evaluating options:
The trend we’re seeing at Stitch (where I’m co-founder & CEO) is that, in addition to popular data sources like MySQL and Salesforce, more and more people have a few data sources that are much less common. There’s been an explosion in the number of tools that companies use to run their business, with a 2017 survey of the marketing technology landscape finding over 5,000 different products (up from 150 in 2011).
This is obviously important if your current ETL needs include a data source that no commercial ETL vendors support. However, it’s also important to choose a tool that will enable you to incorporate whatever new sources you will need in the future.
To do this, make sure that you choose a vendor that has a great API for importing data (which is true of many of the folks mentioned on this thread). With that, you can always write a script that pushes data to their API to be consolidated with your other data sources.
There are some drawbacks to this approach that you should keep in mind:
- If you’re reading this thread, you’re trying get away from building ETL scripts in the first place
- You’re on the hook for maintaining these scripts, which is a much bigger job than the initial build
- You then have to do the operational work to make sure that these scripts run indefinitely and notify the right people when there is a problem with authorization, data volume, API quotas, etc. This is the biggest job of all.
This is why we launched Singer.io, an open source project for ETL integrations. There are more than 20 open source integrations to data sources (or “taps” as we call them), and more are being built all of the time. If you’re building a new one, you can reuse code from the existing taps and helper utilities. We also have partners who can build custom taps.
These taps are maintained by Stitch and the community, so you don’t have to bear that burden on your own. Finally, we can incorporate any integration written to the Singer specification directly into our system, so you can run Singer taps on your own hardware or from within Stitch.