Today's businesses aim to become "by-the-second" organizations that can react swiftly to changes in their supply chain, inventory, customer behavior, and other domains. Whether through online checkout or customer service encounters, they also strive to provide exceptional client experiences. Real-time information should be available to all companies, regardless of size or budget, and it should be integrated into a unified data platform so that everything works together. We're making great strides in helping businesses accomplish these objectives with today's preview release of BigQuery Engine for Apache Flink.
Apache Flink BigQuery Engine
Build and run real-time streaming applications with a fully managed Flink service connected to BigQuery.Qualities
Use real-time data to update your AI and unified data platform
Make business decisions based on real-time insights by utilizing a scalable and integrated streaming platform that is based on the popular Apache Flink and Apache Kafka technologies. When combined with Google's special AI/ML capabilities in BigQuery, you can make the most of your data. You can scale effectively and iterate rapidly without being limited by infrastructure management thanks to the integrated security and governance.To save time and money, use a serverless Flink engine
To take advantage of real-time data, businesses utilize Google Cloud to create streaming applications. However, they are often burdened by the operational strain of managing self-managed Flink, optimizing many configurations, meeting the demands of several workloads while keeping costs under control, and keeping up with changes. BigQuery Engine for Apache Flink's serverless architecture reduces this operating burden and allows its customers to focus on their key skills, such as business innovation.Compliant with the open-source Apache Flink project
BigQuery Engine for Apache Flink enables the lifting and migration of existing streaming applications that use the free source Apache Flink framework to Google Cloud without requiring code changes or reliance on external services. Combining Google Cloud with Google Managed Service for Apache Kafka (now GA) makes it easy to modernize and migrate your streaming analytics.Simplifying ETL
ETL streaming for your AI-ready data platform
You can ingest data streams from sources like Kafka, perform transformations, and then instantly load them into BigQuery for analysis and storage with Apache Flink's open and flexible framework for real-time ETL. Faster data analysis and decision-making are made possible by the benefits of open source extensibility and adaptability to different data sources.Create event-driven applications
Event-driven apps help companies with fraud detection models, recommendation engines, and marketing customisation, among other things. Real-time event streams from many sources, including user activity and payments, can be captured using Google Cloud's managed Apache Kafka service. The Apache Flink engine then processes these streams with low latency, enabling complex operations like real-time processing.Create a platform for real-time data and AI
The BigQuery Engine of Apache Without worrying about infrastructure maintenance, you can use Flink for stream analytics. To evaluate data in real time, use Flink's SQL or DataStream APIs. Create dashboards by streaming your data to BigQuery and connecting it to visualization tools. Utilize Flink's libraries to monitor work performance and stream machine learning.BigQuery Engine for Apache Flink's state-of-the-art real-time intelligence platform allows users to:
- Make use of the popular streaming technology offered by Google Cloud. BigQuery Engine for Apache Flink makes it easier to lift and migrate existing streaming applications that use the open-source Apache Flink framework to Google Cloud without requiring code changes or reliance on third-party services. Combining Google Cloud with Google Managed Service for Apache Kafka (now GA) makes it easy to modernize and migrate your streaming analytics.
- Reduce the operational burden. BigQuery Engine for Apache Flink reduces operational load and frees their clients to focus on their core skills innovating their businesses because it is entirely serverless.
- Provide real-time data to AI. Enterprise developers working with gen AI are looking for a scalable and well-integrated streaming platform that is based on the popular Apache Flink and Apache Kafka technologies and can be paired with Google's special AI/ML capabilities in BigQuery.
Many real-time analytics innovations have been made available to Google Cloud customers with the release of BigQuery Engine for Apache Flink. These include Dataflow Job Builder, which helps users define and implement a streaming pipeline through a visual user interface, and BigQuery continuous queries, which let users use SQL to analyze incoming data in BigQuery in real-time.
Thanks to BigQuery Engine for Apache Flink, Google's cloud streaming service now offers the well-known open-source Flink and Kafka systems, SQL-based, simple streaming with BigQuery continuous queries, and advanced multimodal data streaming with Dataflow, including support for Iceberg. Together with these features, BigQuery connects your data to the best AI tools available, including Gemma, Gemini, and open models.
When your data is real-time, new AI possibilities become available
As we look to the future, it is clear that generative AI has reignited interest in the potential of data-driven experiences and insights. AI works best when it has access to the most recent context, especially creative AI. By combining real-time interactions with past purchase data, retailers may personalize the shopping experiences of their customers. If your company offers financial services, you can use real-time transactions to enhance your fraud detection model. Real-time data combined with AI enables real-time predictions and inferences for your business applications, including integrating tiny models like Gemma into your streaming pipelines, real-time user support through Retrieval Augmented Generation (RAG), and fresh data for model training.Regardless of the specific streaming architecture or streaming engine you use, it is taking a platform approach to offer capabilities across the board to enable real-time data for your future AI use cases. Features like distributed counting in Bigtable, the RunInference transform, support for Vertex AI text-embeddings, Dataflow enrichment transforms, and many more have made it easier than ever to create real-time AI applications.
Google Cloud is excited to give you access to these features and continue to offer you more choices and flexibility when it comes to enabling your unified data and AI platform to operate in real-time data. Learn more about BigQuery Engine for Apache Flink and get started using it in the Google Cloud console immediately.
BigQuery Engine for Apache Flink pricing
| BigQuery Engine for Apache Flink pricing | Usage is billed for resources that your Flink jobs consume, including compute slots and state storage. | |
|---|---|---|
| Service and usage | Description | Price (USD) |
| Compute slot | Compute slots measure Flink application resource usage, billed per second. Usage is displayed in hours for hourly pricing. | Starting at$0.12per hour |
| State storage | State storage rate is a monthly rate. | Starting at$0.04per GiB per month |
0 Comments