AUSTIN, Texas, Dec. 15, 2021 /PRNewswire-PRWeb/ -- Since its spinoff from GitLab in June of this year, the Meltano ELT (extract, load, transform) project has attracted attention from a growing group of upstream contributors and enterprise users. Momentum behind the Meltano DataOps Operating System (OS) underscores the market need for an open source alternative that offers a DevOps-like workflow and a wide array of ELT integrations. Meltano was launched with financial backing from a number of notable venture capital and angel investors including Alphabet's GV.
**The Need for Meltano**
Organizations have data spread across numerous databases and data formats. Combining that data and querying, it can generate useful insights like customer purchasing trends and preferences—insights that would be impossible to extract from individual datasets. To achieve this, data is combined and transformed through ELT into a format suitable for analysis. Meltano is an open source DataOps OS to accomplish this quickly and cost effectively.
Momentum since June for the Meltano project includes:
*Community and User Growth Announced*
Since July, almost 2,000 organizations and individuals have joined the Meltano community. Notable users include HackerOne, Netlify, Remote.com, Vox and Zapier. Meltano users find the project a powerful means to move data at a lower cost than alternative means. It is inexpensive to deploy and scale (open source), and users can control their environments via self hosting. Open source Meltano is adaptable, free and community-driven. Meltano helps users make better decisions by empowering them with insights from their data life cycle and the output of that cycle. Its consistent integration layer and observability across the stack, coupled with best-in-class tooling integration make Meltano the DataOps OS an increasingly popular choice for organizations seeking greater confidence in their data governance.
*Singer Working Group Formed*
The Singer protocol has become the open source standard for data movement. Until recently, there was no cross-functional governing body to guide the development of the Singer standard and create best practices. In October, Meltano announced, along with seven other member companies, support for the future growth of the Singer spec and the Singer ecosystem via the Singer Working Group. Members include:
- AppliedLabs – maintainers of multiple taps and owners of singerly.co, a hosted Meltano and Airflow platform
- AutoIDM – tap/target dev and support, data engineering, and reverse ETL consultancy
- Bytecode – data consultancy, maintainers of over 30 taps on the Hub
- Hotglue – data integration platform which leverages Singer for data movement
- Meltano – creators of Meltano, MeltanoHub, and the SDK for Singer Taps and Targets
- Talend – maintainers of Stitch, original founders of Singer
- Wise – creators of Pipelinewise
- Y42 – no-code data platform, maintainers of multiple Singer taps and targets
*Meltano Labs Launched*
In October, MeltanoLabs was launched as the new home for any Singer connector or Meltano plugin that doesn't otherwise have a public repo. One of the opportunities in the Singer ecosystem was a need to better support a community of users and contributors to maintain connectors.
*Meltano Environments Becomes Public*
In November, Meltano launched Meltano Environments as a way to quickly toggle the execution of commands between different configuration sets by passing a single `MELTANO_ENVIRONMENT` environment variable or the `–environment` CLI option. The utility of this key feature is the ability to bring the DevOps concept of having isolated environments for building, testing and deploying to data tooling.
*Composable Pipelines Available*
Composable pipelines is a new way to run data workflows in Meltano, with installed plugins that can be chained together in new ways. Previously, users were limited to running a tap and target with non-ideal syntax for running dbt afterwards, but composable pipelines dramatically improve the experience of running dbt via the new meltano run command. Meltano now also supports inline transformations (or, stream maps) for manipulating data on the fly between Singer taps and targets. Composable pipelines also enable the chaining of data tests, such as dbt test or Great Expectations (coming soon!) in line with ELT pipelines.
*Meltano SDK for Singer Taps and Targets*
With the Meltano SDK, users can build high-quality connectors for new sources and destinations in a matter of hours.
*MeltanoHub Launched*
MeltanoHub was launched to ease discovery of over 300 open source connectors maintained by the ever-growing Meltano and Singer communities.
**Supporting Quotes**
"Meltano rocks," said Oliver Laslett, co-founder and CTO at Lightdash. "It feels production-grade while still being really easy to configure. I picked up the GitHub tap, customized it slightly for my use case, and it's been running reliably for months."
"Meltano has improved our cycle time greatly by reducing the time to develop and deploy," said Maarten van Gijssel, senior software engineer at HackerOne. "Development has improved because Meltano makes it really easy to run and debug a pipeline. Deployment has improved because all the dependencies are managed by Meltano, making it very convenient to run the entire data stack from a single Docker image. With Meltano, it's easy to set up an integrated environment from your local machine and CI, allowing us to iterate and test a lot faster.'
"Our community has been busy since the spinoff from GitLab in June," said Douwe Maan, founder and CEO at Meltano. "The weekly cadence of code commits and new releases has been made possible by the hard work of the engineering team at Meltano and the focused efforts of our growing community of more than 2,000. Watching our vision come to life has been an incredibly energizing experience, and in the months ahead we'll see more features and capabilities that will make Meltano the DataOps OS of choice for data teams that need greater agility and velocity. Meltano is showing how DataOps, inspired by the best practices of DevOps, is the way to get there."
**Meltano and Singer**
Meltano embraces the Singer standard for data movement and the Singer library of over 250 community-maintained connectors. The project uses dbt for transformation and Airflow for orchestration. Meltano simplifies the use and deployment of these tools to discover existing connectors and build new ones. The Meltano roadmap includes adding support for open source tools like Superset and Lightdash for visualization as well as alternative orchestration solutions like Dagster and Prefect, while continuing to grow its connector library.
**About Meltano**
Meltano enables everyone to realize the full potential of their data. We are building a modular, open source DataOps OS to be the foundation of every team's ideal data stack. Meltano simplifies configuration, deployment and monitoring, unites best-in-class open source tools and technologies for the data lifecycle. It allows data teams to benefit from DevOps best practices such as version control, code review and CI/CD.
Meltano works alongside its community of almost 2,000 data professionals to build the next generation of data infrastructure and support a large ecosystem of open source connectors. Meltano is in use today by organizations including GitLab, HackerOne, Netlify, Remote.com and Zapier. Join our community on Slack and Twitter: https://twitter.com/meltanodata
Media Contact
Cristin Connelly Zegers, Cathey Communications, 404-931-6752, [email protected]
SOURCE Meltano
Share this article