Apache Spark

Apache Spark™ provides the foundation for large-scale data processing and analytics. By integrating Spark Connect and @spice_ai OSS, teams can unlock the speed and concurrency needed for operational AI workloads, including fraud detection, predictive analytics, and real-time…

See Apache Spark + @spice_ai in Action: Faster Queries + Real-Time AI Decisioning 🤝 Apache Spark™ delivers at scale for data processing and analytics. With Spark Connect and Spice AI OSS, teams can also enable low-latency, high-concurrency access to Spark data for operational…

📣 Announcing: Apache Spark™ Python Data Source for @huggingface AI Datasets! During this virtual event, you’ll learn how Apache Spark™ 4.x Python Data Source API allows Hugging Face to extend datasets for AI workloads. Why attend? ✅ 𝗗𝗶𝘀𝗰𝗼𝘃𝗲𝗿 𝘁𝗵𝗲 𝗹𝗮𝘁𝗲𝘀𝘁…

Building Operational AI Apps on Apache Spark? Learn How to Overcome Latency and Concurrency Challenges 🚀 Apache Spark™ is known for large-scale data processing and analytics. With the right integrations, it can also deliver the low-latency performance needed for operational AI…

Can Apache Spark™ power operational AI workloads? Find out on July 30!  🚀 Apache Spark™ is the gold standard for large-scale data processing and analytics. With additional integrations, it can handle low-latency operational AI workloads—think real-time recommendations, fraud…

[ANNOUNCEMENT] The Apache Spark 4.0.0™ release is here! 🎉 Congrats to the #ApacheSpark community! This milestone is a testament to tremendous collaboration, resolving over 5100 tickets with contributions from more than 390 individuals. Try it out! ➡️ spark.apache.org/releases/spark…

🚀 The upcoming Apache Spark™ 4.0.0 release introduces the ANSI SQL/PSM standard & extends the current Spark SQL functionality to include: ✅Procedural logic & control flow ✅Complex data transformations ✅Familiar scripting for analysts ⬇️ Learn more! linkedin.com/pulse/sql-scri…

[ANNOUNCEMENT] Congrats to the Apache Spark community and all the contributors! The Apache Spark 3.5.0 release is here. Try it out! spark.apache.org/releases/spark…

#ApacheSpark 3.4 is the fifth release of the 3.x line which resolved > 2,600 lira tickets! One of the highlighted features is the #python client for #sparkconnect Want to know more? Check out Martin Grund's @PyConDE session pretalx.com/pyconde-pydata…

[ANNOUNCEMENT] Congrats to the Apache Spark community and all the contributors! The Apache Spark 3.4.0 is here. Try it out! spark.apache.org/releases/spark…

In his #DataAISummit 2022 keynote - @rxin highlighted #SparkConnect which introduces a decoupled client-server architecture for #ApacheSpark that allows remote connectivity to Spark clusters. Now, we have our first PR! github.com/apache/spark/p…

[ANNOUNCEMENT] Congrats to the Apache Spark community and all the contributors! The Apache Spark 3.3 is here. Try it out! spark.apache.org/releases/spark…

Very excited that @ApacheSpark won the SIGMOD System Award this year. Congrats to the whole community behind the project!

Some great details on how #ApacheSpark handles task retries in #StructuredStreaming

How does Apache Spark handle task retries in #StructuredStreaming? That's the question I tried to answer in the new blog post 👉 waitingforcode.com/apache-spark-s…

What's new in Apache Spark 3.2.0 - PySpark and Pandas by @waitingforcode waitingforcode.com/pyspark/what-n…

[ANNOUNCEMENT] Congrats to the Apache Spark community and all the contributors! The Apache Spark 3.2 is here. Try it out! spark.apache.org/releases/spark…

[ANNOUNCEMENT] Congrats to the Apache Spark community and all the contributors! The Apache Spark 3.1 is here. Try it out! spark.apache.org/releases/spark…

[ANNOUNCEMENT] We are happy to announce the availability of Spark 3.0.2! This is a maintenance release containing stability fixes, based on the branch-3.0 maintenance branch of Spark. View the release notes for more info. spark.apache.org/releases/spark…