Databricks Donates Declarative Pipelines to Apache Spark™ Open Source Project
Spark Declarative Pipelines provides an easier way to define and execute data pipelines for both batch and streaming ETL workloads across any Apache Spark-supported data source, including cloud storage, message buses, change data feeds and external systems. This battle-tested declarative framework for building data pipelines helps engineers address common pain points like complex pipeline authoring, manual operations overhead and siloed batch/streaming.
Spark Declarative Pipelines is based on Databricks' core declarative ETL framework, which is used by thousands of customers. With the proven ability to handle complex data engineering workloads and low-latency streaming, Spark Declarative Pipelines lays the foundation for the next generation of data processing and governance. With Spark Declarative Pipelines, more community members can begin to cut engineering time and costs and reliably support new AI agent systems and other workloads in production.
"Our commitment to open source is unwavering. With origins in academia and the open source community, Databricks was founded in 2013 by the original creators of the lakehouse architecture and open source projects including Apache Spark, Delta Lake, MLflow and Unity Catalog," said Matei Zaharia, Co-founder and CTO of Databricks. "We worked closely with the community to help remove friction around data formats that kept information siloed. Spark Declarative Pipelines now gives enterprises an open way to build high-quality pipelines."
Key benefits of Spark Declarative Pipelines include:
Simplifying pipeline authoring: Data engineers and analysts can quickly declare robust pipelines with minimal coding, focusing on delivering business-critical insights.
Improved operability by design: Spark Declarative Pipelines help catch issues earlier in development through clear pipeline definitions that are validated in full prior to execution, reducing the risk of failures downstream and making pipelines easier to troubleshoot and maintain.
Unified batch and streaming: Data teams can flexibly meet both real-time and periodic processing needs through a single API for defining and managing batch and streaming data pipelines, simplifying development and maintenance.
"Declarative pipelines hide the complexity of modern data engineering under a simple, intuitive programming model. As an engineering manager, I love the fact that my engineers can focus on what matters most to the business. It's exciting to see this level of innovation now being open-sourced, making it accessible to even more teams." — Jian (Miracle) Zhou, Senior Engineering Manager, Navy Federal Credit Union
"At 84.51˚ we're always looking for ways to make our data pipelines easier to build and maintain, especially as we move toward more open and flexible tools. The declarative approach has been a big help in reducing the amount of code we have to manage, and it's made it easier to support both batch and streaming without stitching together separate systems. Open-sourcing this framework as Spark Declarative Pipelines is a great step for the Spark community." — Brad Turnbaugh, Sr. Data Engineer, 84.51°
About Databricks
Databricks is the Data and AI company. More than 15,000 organizations worldwide — including Block, Comcast, Condé Nast, Rivian, Shell and over 60% of the Fortune 500 — rely on the Databricks Data Intelligence Platform to take control of their data and put it to work with AI. Databricks is headquartered in San Francisco, with offices around the globe and was founded by the original creators of Lakehouse, Apache Spark™, Delta Lake, MLflow, and Unity Catalog. To learn more, follow Databricks on X, LinkedIn and Facebook.
Hashtags
Try Our AI Features
Explore what Daily8 AI can do for you:
Comments
No comments yet...
Related Articles
Cision Canada
17 hours ago
- Cision Canada
WSIB and OCEU reach tentative agreement to end labour disruption Français
TORONTO, July 5, 2025 /CNW/ - The Workplace Safety and Insurance Board (WSIB) is pleased to announce a tentative agreement has been reached with the Ontario Compensation Employees Union (OCEU). The organization looks forward to welcoming back its full team within 24 hours of a successful ratification vote. "Our number one priority has always been—and continues to be—helping the people who depend on us," said Jeff Lang, President and CEO of the WSIB. "We do work that makes a difference in people's lives and in our communities. I am proud of our team's work the last few weeks and am very excited for everyone to come back together so we can keep supporting Ontarians who need us." As normal service resumes, people can continue to log in to the WSIB's website anytime, 24/7 to: Report an injury or illness Submit documents for an existing claim See claim, payment, and health benefit information in real time Register a new business Access clearances "We thank Ontarians for the patience they've shown us during the labour disruption," said Lang. "Our team is committed to delivering better, easier, and faster service to Ontarians and we can't wait to get back to doing what we do best – helping people." The WSIB provides workplace injury and illness insurance to more than 5.3 million people in more than 300,000 workplaces across Ontario. When an injury or illness happens on the job, the WSIB provides wage-loss benefits, medical coverage and support to help people get back to work. Funded by Ontario businesses, the WSIB also provides no-fault collective liability insurance and access to industry-specific health and safety information.
Cision Canada
20 hours ago
- Cision Canada
Sheraton Vancouver Guildford workers prepare to strike
SURREY, BC, July 5, 2025 /CNW/ - Unifor Local 3000 members at the Sheraton Vancouver Guildford Hotel are preparing for strike action later today. "The employer has left us with no other choice," said Unifor Western Regional Director Gavin McGarrigle. "Our members are the heart of the hotel, and after more than a year without a fair agreement, they are prepared to strike to demand respect and fair wages." Unifor Local 3000 members voted to reject a tentative agreement with the company. The union will be in a legal strike position as of 12:45 p.m. Pacific on July 5, 2025. The bargaining unit includes 120 full-time and 24 part-time workers, who have been without a contract since April 2024. The Sheraton Vancouver Guildford is owned and operated by Pacific Reach Properties, a Vancouver-based investment firm with diverse holdings across real estate, hospitality, healthcare, and tech sectors. The company has sub-contracted human resources and collective bargaining responsibilities to Hospitality Industrial Relations (HIR), a Vancouver-based multi-employer group that negotiates on behalf of various hotel and tourism operators in British Columbia. "Wages at the Sheraton have failed to keep up with the rising cost of living in the Lower Mainland," said Adrian Burnett, President of Unifor Local 3000. "Our members are struggling to make ends meet while working for a hotel owned by a company with deep pockets. It's unacceptable." Unifor members voted overwhelmingly in favour of strike action if necessary, sending a strong message to the employer that a fair deal must be secured. The union remains committed to negotiating an agreement at the table but is prepared to take strike action should an agreement not be reached. Unifor is Canada's largest union in the private sector, representing 320,000 workers in every major area of the economy. The union advocates for all working people and their rights, fights for equality and social justice in Canada and abroad, and strives to create progressive change for a better future.
Cision Canada
a day ago
- Cision Canada
Lotto Max - A $65 million jackpot and an estimated 8 Maxmillions in the next draw Français
MONTRÉAL, July 5, 2025 /CNW/ - In the next Lotto Max draw, slated for Tuesday, July 8, the prize pool will be $73 million and include a $65 million jackpot and an estimated 8 Maxmillions. "A Lotto Max jackpot was just won in June. We're hoping that Lady Luck strikes again in Québec in Tuesday's draw, when the jackpot is $65 million," says Isabelle Jean, Executive Vice-President and Chief Operating Officer of Lottery Games at Loto-Québec. Lotto Max continues to churn out winners in Québec Lotto Max players in Québec have won over $3.925 billion since its launch in 2009. In total, there have been 46 jackpot wins in the province, including 3 of $70,000,000—the largest jackpots ever won in Québec—in the February 25 and October 9, 2020, and June 7, 2022, draws. One $65,000,000 jackpot, four $60,000,000 jackpots, five $55,000,000 jackpots, and nine $50,000,000 jackpots have also been won in Quebec. In addition to these jackpot wins, Lotto Max players in Québec have also won a total of 247 Maxmillions prizes, each worth $1 million, and 151 shares of Maxmillions prizes. Draw results are available online at About Loto-Québec Loto-Québec has been delivering entertainment to Quebecers for over 50 years, and its operations benefit Québec as a whole. Last year, the corporation paid out almost $1.8 billion in prizes to winners across Québec. Major prize winners are listed on the Winners page in the Lotteries section of Read their stories. The responsible commercialization of lottery and gaming products is central to Loto-Québec's operations. The corporation has been granted the highest internationally recognized certification in responsible gaming by the World Lottery Association.
