Resources
Blog
What are WebSockets and how do they work?
March 23, 2023
WebSockets are a protocol that allows for real-time, two-way communication between a client and a server over a single, long-lived connection. WebSockets have gained popularity in recent years due to their ability to provide low-latency, real-time communication in web applications. In this article, we’ll explore what WebSockets are, how they work, and some common use …
Is your sportsbook prepared for Cheltenham?
March 15, 2023
The world of sportsbooks is on an ever-increasing rise with records being set and broken regularly. Some sportsbooks have reported taking 50,000 bets per minute during the peak of the Superbowl. This year already looks set to break previously set records. Betsson has reported that last year was a very good year for them in …
DiffusionData nominated for "Best Cloud Platform for Trading Applications" award
March 01, 2023
DiffusionData has been nominated for the “Best Cloud Platform for Trading applications” award at the TradingTech Insight Awards USA 2023. We have been selected by an independent advisory board in the USA and voting has now opened to pick the winners of each category! About the Awards: These annual awards recognise excellence in trading solutions …
Market Data in the Cloud
January 18, 2023
By Riaz Mohammed, CTO DiffusionData Overview and basic concepts Market data ecosystems are typically hosted on premise. Market Data Types: Pre-trade and Post-trade Market data. Containing Bid and Ask for various instruments – essential for trading. Market Data is used for real-time trading, risk analysis, analytics and more. Trading uses real-time Market data …
Award winning year for DiffusionData
January 09, 2023
It is always wonderful when the industry recognises the product and the hardwork of the many who works behind the scenes. DiffusionData, leader in real time data streaming has had an excellent 2022 in this regard winning 4 prestigious awards and numerous accolades for its Intelligent Data Platform. The wins included “Most innovative data-driven transformation …
Diffusion For the Hyper Connected World
December 24, 2022
The latest news, views, and announcements By Riaz Mohammed, CTO DiffusionData Our world is getting more and more hyper-connected and there is no sign of any slowdown. Those who were resisting being digital, were forced during covid. As the world becomes increasingly comfortable with technology, we can safely say, almost everyone and most devices, are connected …
Simplifying Real-time Data with Diffusion
December 20, 2022
The latest news, views, and announcements By Riaz Mohammed, CTO DiffusionData Today’s enterprises need access to real-time data and insights for operational efficiency and more importantly to be competitive. Businesses are focused on information management for distributed data sources such that they can provide best in class user experience to customers through frictionless access to …
Will the FIFA World Cup be record-breaking for sportsbooks?
November 21, 2022
With the FIFA World Cup starting this week the world of sportsbooks looks set to break previously set records. Betsson has reported that due to the early start of the European football leagues they have recorded back-to-back best-ever quarters so far this year. The World Cup looks set to propel sportsbooks past previously set records. …
Integration with Diffusion: The Gateway Framework
May 30, 2022
Release 6.8 of the Diffusion Intelligent Data Platform for real-time applications, supports a powerful new feature called the ‘Gateway Framework’. What is the Gateway Framework? For many years Diffusion has been a leading-edge technology product concentrating on the fast and efficient caching and delivery of data to diverse clients, such as web, mobile, IoT, etc. …
Topic Views in Release 6.8 – Conditionals and Calculations
May 16, 2022
Release 6.8 of the Diffusion Intelligent Data Platform for real-time applications includes new additions to the popular data-wrangling feature called ‘topic views’. It is now possible to include conditional processing of JSON topic values, and also to perform calculations on numbers within the JSON and inject the results into the output values. What’s new in …
How to write good APIs
March 03, 2022
See why the secret to writing a successful API is to model what the product does and not how the product works. …
CDC (Change Data Capture) Adapter
November 22, 2021
Introduction CDC (Change Data Capture) is a process of identifying and tracking data change events in databases. CDC is an efficient mechanism to achieve reliable and scalable data replication across different systems. In the 6.7 release of Diffusion™, we have introduced a brand-new CDC adapter, which enables users to replicate data from databases into a …
Session Trees - A New Feature for 6.7
October 18, 2021
Release 6.7 of Diffusion™ introduces a new and exciting feature called ‘session trees‘. Some History – Routing Topics (You can skip this section if you are not a current user of routing topics). Existing users of Diffusion may be familiar with ‘routing topics’. Routing topics have been a part of Diffusion for some time and …
Using Diffusion with Docker Volumes
October 05, 2021
In our tutorial on Using Diffusion with Docker, we saw how to pull a Diffusion image, start the server, and use a Dockerfile to build your own image on top of Diffusion. In this tutorial, we will look at using Docker Volumes to enable custom configuration and persistence on your Diffusion server, without the need …
Diffusion Adapter Management and Monitoring
August 12, 2021
With the 6.7 release of Diffusion™, all the included data adapters (the Kafka adapter, the JMS adapter, and the new CDC adapter) can be monitored and controlled from the Diffusion Management Console of the connected server, or any other server in the same cluster. Once started, each adapter is visible in the console ‘Network’ tab. …
Sportsbook Demo: Now in MQTT Flavor
March 24, 2021
MQTT is a publish-subscribe messaging protocol commonly used to connect Internet of Things (IoT) applications. With the release of Diffusion 6.6, Diffusion supports clients connecting with MQTT 5.0. See the 6.6 manual for full details. A wide range of IoT devices can connect directly to Diffusion, send and receive data, interact with other Diffusion clients, …
Extra Topic View Features in the Diffusion 6.6 Release
March 24, 2021
Diffusion™ 6.3 introduced topic views, a mechanism to produce virtual topics (known as reference topics) that take their values from other topics in the topic tree. In 6.4, we added topic view expansion, enabling a single source topic to be expanded to produce many derived reference topics. In Diffusion 6.5, the concepts of fan-out and …
Fine-Grained Fan-out and Replication of Kafka Event Firehose Between Clusters
January 08, 2021
Fine-grained fan-out and replication of Kafka event firehose between clusters/sites. Introduction to Diffusion Real-Time Event Stream through a simple application using Diffusion Cloud and Apache Kafka. This simple project illustrates real-time replication and fan-out of foreign exchange (fx) event streams from Kafka cluster A to Kafka cluster B, through Diffusion Cloud instance via the use …
Optimize MQTT performance with DIffusion®
January 04, 2021
MQTT is a publish-subscribe messaging protocol commonly used to connect Internet of Things (IoT) applications. A wide range of IoT devices can now connect directly to Diffusion, send and receive data, interact with other Diffusion clients, and benefit from Diffusion’s event data processing features and rich security framework. MQTT can also be used as a …
Diffusion Now Supports MQTT
December 10, 2020
Support for the OASIS MQTT open standard protocol is the main feature added to Diffusion 6.6 Preview 2, the latest release of the Diffusion® Intelligent Event Data Platform. MQTT is a publish-subscribe messaging protocol commonly used to connect Internet of Things (IoT) applications. A wide range of IoT devices can now connect directly to Diffusion, …
Extend Kafka Topic Modeling with Diffusion
November 24, 2020
Stream through an application using Diffusion Cloud and Apache Kafka during this walkthrough of projects that illustrate production and consumption of foreign exchange data. …
New Kafka Adapter: Connect Diffusion 6.6 with Apache Kafka
November 17, 2020
Check out Diffusion 6.6 Preview 1. A beta version of the new Kafka adapter, it’s available for on-premise Diffusion and translates data between Diffusion topics and Apache Kafka events. …
Time Series Topic Enhancements in Diffusion 6.6
November 17, 2020
For Diffusion 6.6, we are evolving the time series topic APIs to deliver a streamlined development experience. This post will summarize some of the improvements coming to 6.6. All these improvements are also available in Diffusion 6.6 Preview 1, so you can try them right now! Time series topics: a refresher Time series topics are …
Merge Topic Data with Topic View Inserts in Diffusion 6.6
October 23, 2020
Diffusion™ 6.3 introduced topic views, a mechanism to produce virtual topics (known as reference topics) that refer to other topics in the topic tree for their value. In 6.4, we added topic view expansion, enabling a single source topic to be expanded to produce many derived reference topics. In Diffusion 6.5, the concepts of fan-out …
How to build a Real-Time Messaging App using Diffusion
September 15, 2020
Real-Time Messaging Introduction to Diffusion Real-Time Messaging through a simple application using Diffusion Cloud. A set of simple projects, illustrating production and consumption of messages to and from a Diffusion Cloud instance. These JavaScript code examples enable users to interact in real-time across various rooms or topics. You can also use other programming languages from our SDKs, …
Diffusion 6.5 Is Released: Build Distributed, Scalable Applications with Enterprise-Grade Access Control
July 15, 2020
We’re excited to announce that the latest update to the Diffusion™ Intelligent Data Mesh is now available, both for download and as a Diffusion Cloud service. Diffusion 6.5 is full of ground-breaking features that make it easier than ever to develop and maintain advanced real-time applications. We’ve been guided in our development by the real-world …
Security Changes in Diffusion 6.5
July 10, 2020
Controlling access to data Diffusion™ excels at delivering real-time data, but it’s important to have control over exactly where your data gets delivered. In a typical deployment, your Diffusion server accepts connections over the Internet from multiple remote users and applications. For most applications, you trust some users more than others, so you need to …
The Diffusion Management Console, Past and Present
July 10, 2020
The Diffusion Management Console has been a staple feature of Diffusion since 2015, allowing users to monitor and, more recently, configure their Diffusion server. It has undergone a lot of change since its inception. With the release of Diffusion 6.3 in May 2019, we began to focus on making the console a capable tool not …
Remote Topic Views – Beyond Fan-out
July 09, 2020
Diffusion™ has long had the ability to replicate topics from an upstream (or primary) server to one or more secondary servers using the fan-out feature. Fan-out is configured on the secondary server, enabling selected topics to be replicated from the primary server to the secondary. As those topics update on the primary server, the updates …
Introducing the Diffusion Cloud REST API
July 09, 2020
Publish and Request Real-Time Data for Tens of Thousands of App Users or Sensors Via Simple HTTP in the Programming Language of Your Choice Diffusion Cloud is used successfully by customers around the world to send billions of real-time messages each month. One of the most common type of feature requests we receive is …
More Cluster-Aware Features In Diffusion 6.4 and 6.5
May 14, 2020
When your application delivers real-time data, high availability is important. In production, you will probably want to take advantage of Diffusion™’s clustering capabilities. By clustering multiple Diffusion servers, you can add resilience to your application and keep your data flowing even through a hardware failure. What is cluster awareness? In earlier versions of Diffusion, not …
Using Promises with the JavaScript Diffusion Client
March 06, 2020
Many functions in the JavaScript Diffusion client return a Result object. This object can be used to obtain the return values of asynchronously executed code. …
New Topic View Features in 6.4
March 02, 2020
Diffusion 6.3 introduced topic views, a mechanism to produce virtual topics, that refer to other topics in the topic tree for their value. Topic views are an evolution and generalization of slave topics. A slave topic is derived from a single source topic in the topic tree. Sessions may subscribe to the slave without having …
Expanded Feature Support In The Diffusion C Client
December 18, 2019
Diffusion 6.1 and 6.2 added many functions and capabilities to the Diffusion C client that were previously only available in other languages: Adding topics through topic specifications (added in 6.1) Support for string, int64, double and recordV2 topic types (added in 6.1) Value stream API support (added in 6.1) Enhanced topic update (added in 6.2) Request/response …
Security and Session Properties Enhancements in Diffusion 6.2
December 06, 2019
Diffusion 6.2 introduces significant improvements to security features and the use of session properties. These include additional features for managing security roles and more control over the use of session properties in authentication and session filtering. Session properties are values assigned to every client session in the form of key/value pairs. There is a set …
New Fetch API in Diffusion Release 6.2
November 27, 2019
Version 6.2 of the Diffusion Intelligent Data Platform introduces a major improvement in the form of the new fetch API. In Diffusion terms, ‘fetch’ refers to the operation that allows a client to request the current state of a topic or topics without subscribing to those topics. The new fetch API is a powerful interface …
New Update API in Release 6.2
November 20, 2018
One of the key abilities in Diffusion is the ability to update the value of topics. Updating allows new values to be provided for topics. In Diffusion 6.2 a new update API has been introduced. This new API introduces new ways of coordinating multiple updaters, handling missing topics and a safer interface. There are five main changes …
Diffusion 6.2 has been released
November 13, 2018
The latest update to the Diffusion™ Intelligent Data platform is now available. There are a number of exciting new features in Diffusion 6.2, allowing users to simplify development of multi-user and collaborative applications; control security policies with greater flexibility; fetch and stream large data sets; and apply data compression to all topic messages. New Platform …
Introducing the New Diffusion 6.1 .NET Client Library
June 18, 2018
We introduced a new Diffusion .NET Client Library that shipped with Diffusion 6.1. The new version of the client library allows you to develop .NET Diffusion clients for Windows, macOS and Linux thanks to the .NET Standard 2.0. New library and NuGet package With Diffusion 6.1, we introduced a new .NET Client Library and NuGet …
Diffusion 6.1 introduces powerful new features for dynamic topic management
April 27, 2018
In Push Technology’s ongoing effort to simplify data management, optimization, and integration for companies developing business critical applications, Diffusion 6.1 introduces a pair of powerful new features: automatic topic removal and topic ownership. Introduction Diffusion uses topics to deliver data to clients via a publish and subscribe interaction model. A Diffusion topic is an identified stream …
Getting started with JSON & Binary in the JS client
April 17, 2017
With the release of Diffusion 5.7 two new topic types have been introduced; JSON & Binary. The former is especially useful for users of the JavaScript client, since JSON is the defacto standard for handling data in web-applications and enjoys native support in JS. This article will go through some simple examples of using these new topic types, and …
Reliable Reconnect
November 21, 2016
DiffusionTM communication is composed of discrete messages. Each client sends various types of message to its server, and separately receives messages from the server. There are messages that carry new values for subscribed topics; application messages sent using the messaging API; and service messages that implement API features. All Diffusion’s protocols assure message delivery while a client …
Building the Usage Based Insurance App – pt 2
November 01, 2016
Following on from part 1 of this blog series, we’ll next look at how the demo is constructed, the Bluemix services we used, and how they interact with each other. As mentioned previously, there are several IBM services in use alongside our own Diffusion Cloud service – IBM’s Tradeoff Analytics, IBM’s Streaming Analytics, and IBM’s Cloudant …
Building on Value Streams
September 16, 2016
In DiffusionTM 5.7 we introduced data types, value streams, Binary topics and JSON topics. Together they streamline much of the work that previously would have required application code, but they still don’t do everything. Value streams are an enhanced version of topic streams that receive typed values instead of Content objects that might contain a delta to a value. …
Meet the Diffusion C API
September 08, 2016
A recent release of Diffusion Cloud added C to the list of APIs we provide for users of our cloud service. We’re pretty excited about this and here’s why. C is one of the most versatile and enduring programming languages. It’s not hard to see why: it’s universally available, has no platform or vendor lock-in, it’s …
Diffusion and Diffusion Cloud Security: System Authentication
August 11, 2016
This article is about the system authentication handler in Diffusion and Diffusion Cloud. It is part of a series of articles covering Diffusion and Diffusion Cloud security features. As usual, most of what you will read here applies equally to Diffusion and Diffusion Cloud. In the previous article, we saw how to control who can …
Diffusion and Diffusion Cloud Security: Authentication Providers
August 04, 2016
This is the first in a series of articles covering security features first introduced in Diffusion 5.0, and extended in later releases. I’ll assume Diffusion 5.5 or later because of the declarative authorization features added in that release. Most of what you will read here also applies to Diffusion Cloud. The focus of this article …
Why you Should Care About JSON Efficiency
July 19, 2016
We recently introduced some new functionality in our products to improve the efficiency of your applications when handling JSON – but we thought it would be useful to explain why it’s important, and why you care! JSON has opened many doors for developers, making it simpler for apps to exchange data over the web and …
How to Integrate MongoDB with Diffusion Cloud
July 13, 2016
In this blog, we talk about integrating, Configuring MongoDB for replication with Diffusion Cloud. Read the article to learn more! …
Application Architecture Calls for Reactive, Realtime Systems
July 12, 2016
Application architecture and integration requirements have undergone a sea of change over the past few years. The number of end-users, connected devices, and applications are increasing exponentially. All the while, enterprise data resources, systems of record and computing infrastructure are pushed out to an array of new cloud-based services that are connected – yet separated …
Connection Cascading in the JavaScript Unified Client
July 06, 2016
In Diffusion 5.7, we added a polling (XHR) transport to the existing WebSocket transport. XHR is short for XMLHttpRequest which is an API provided by the browsers for a server and a client to exchange data. A new option, ‘transports’, that the JavaScript client can use to pass in a list of transports to connect to the server, for example: diffusion.connect({ transports …
JSON and Binary Topics for Real-Time Messaging
June 27, 2016
In DiffusionTM 5.7 some new topic types have appeared, namely JSON and Binary. More generally these are referred to as the first examples of universal topics. In this article I will describe what these universal topics are and why they represent a huge step forward for Diffusion users. Everything discussed here also applies to Diffusion Cloud. …
Byte Based Billing: Pricing to Disrupt the Realtime Market
June 15, 2016
With the release of Diffusion Cloud, Push Technology has introduced byte based billing. Data delivered to Diffusion Cloud for publication and the data published on to end user apps are charged at a flat rate of $1.99 per GB. This approach better suits our customers needs. With byte-based billing, our customers will truly pay for …
New Realtime Messaging Features with Diffusion 5.7
June 14, 2016
Diffusion 5.7, a market-proven realtime messaging platform designed for enterprise, has just been released. Find out what’s new so you can benefit from efficient realtime data distribution that optimizes data delivery for your web, mobile and IoT apps. The latest features include JSON/binary delta streaming and enhanced information security. The patent-pending features make live data …
Real-Time Messaging Solution – Kafka vs Diffusion
June 01, 2016
Apache Kafka has grown in prominence since it was originally developed by LinkedIn back in 2011 and a number of notable companies are now using the solution including, Netflix, Spotify and Uber. Apache Kafka is described as a “high-throughput, distributed, publish-subscribe messaging system” and according to an article in datanami, “Open source projects like Kafka …
Ask Engineering: What Tech Talk Interested or Inspired You?
May 11, 2016
In the development team, we’re always sharing cool links and videos with each other. I’ve been introduced to a lot of interesting subjects and ideas in my time at Push. So I knew when I asked the question “What tech talk interested or inspired you?” that I’d get a wide range of engrossing and sometimes …
Building Resilient Applications
April 22, 2016
Many of our objectives here at Push are intended to improve the stability and resilience for applications. An objective shared by the Reactive Manifesto: “[Resilient Apps remain] responsive in the face of failure. This applies not only to highly-available, mission critical systems — any system that is not resilient will be unresponsive after a failure. …
The True Cost of Moving Data
April 05, 2016
Cloud storage is cheap and gets cheaper every day. In January, AWS announced yet another price cut (its 51st to be exact) for select instances. Google hit back criticizing the cut, however even with the January price cuts, Google Cloud Platform is between 15-41 percent less expensive than AWS across several comparable instance types. That’s …
Ask the Office: What Are you Reading?
March 31, 2016
Ever on the lookout for book recommendations, especially for interesting and educational works, I asked our London office what non-fiction or technical book they are reading at the moment. These are some of the answers I got: Martin is reading Flash Boys: A Wall Street Revolt by Michael Lewis Martin says: “This book was recommended …
How to Exploit Event Driven Data
March 18, 2016
Event-driven data is data that is sent or received based on something that happens. For computer programming, that could be a click of a mouse, typing, etc. For apps, it can be similar, but we focus on event-driven data as something that happens outside the app, that impacts what the user will do e.g. there …
Understanding Server Sent Events vs. WebSockets
March 09, 2016
The exploding world of real-time data integration has led many to ask what technologies represent the best choice for their application architecture and how real-time data delivery platforms can help. There are several competing vendors – but not all are created equal. In this post, we’ll discuss the differences between Server-Sent Events and the WebSocket …
REST vs. Push Technology’s Real-Time Data Delivery Solution
March 02, 2016
We have strong opinions here at Push on when REST APIs can be harmful to an app – namely how it can impact a system’s scale, throughput and performance. While REST APIs are tried and trusted for downloading web pages, most mobile and web app users are used to having information delivered to them proactively …
Event-Driven Microservices – Data Integration & Architecture
February 18, 2016
In 1967, Melvin Conway published a paper that asserted system design often reflected the communication structure of the organization building said system. This often meant that larger co-located teams developed monolithic systems, while smaller distributed teams built more loosely coupled systems. However, Conway also found that the initial design of any system was almost never …
When is Your Enterprise Software Solution Ready to go into Production?
February 11, 2016
One of the biggest questions when building an enterprise software solution is whether that solution is ready to go into production. When is enough testing enough? When have all the risks been mitigated? When are you ready? Diffusion is a powerful ingredient software, but it is only one part of a larger solution that can …
Decoupling Your Application Architecture
February 05, 2016
Application or enterprise architecture has become a fast moving world of late, with many voices talking loudly and often about technology choices, integration patterns, standards, etc. Yet throughout all this noise, one simple rule prevails – you must limit coupling between services, and especially between frontend apps and backend resources. Unfortunately, we often fail to …
Replication Update, Smoothing the Rough Edges (pt. 4)
January 27, 2016
Since my last post about Diffusion’s support for high availability (HA) we have made substantial changes to how we replicate data in our latest release, Diffusion 5.6. I’ve already written a little about session replication, topic replication and how we use Hazelcast so I will focus on updating you with the latest changes. We’ve done the …
The diffusion-maven-plugin
January 22, 2016
Here at Push Technology, Java is at the heart of everything we do so we’re really interested in tools and technologies that improve our Java productivity. If you care about Java, you probably have some interest in Maven and how it can be used to facilitate better build and test of your code. (Yes, all …
Almost Everything About Replication – Hazelcast Replication (pt. 3, Hazelcast)
January 23, 2015
And now my third blog post on replication – I hope you liked the last two. This will be the last, at least until some new replication changes are made (writing these posts has given me some great ideas). In this post I am going to talk about Diffusion’s implementation of replication using Hazelcast as a provider. This …
Push Notifications vs. Push Technology’s Diffusion
November 21, 2014
Last week, while at Apps World, one of the most common questions that we were asked was ‘You’re called Push Technology, so do you just do push notifications?’. As Apps World is an expo for multi-platform developers, this is an understandable assumption to be made. But the answer is no, our technology, Diffusion, is an …
Reactive Web Apps – Building a Simple Reactive Web Application
August 13, 2014
With all the buzz about “Reactive” programming, we decided to try our hand at building a simple Reactive web-app for this year’s Devoxx conference. In this blog post, I’ll talk about how we approached the development of the front-end and how we incorporated Reactive principles. The Demo The idea was to create a web-app that …
Emacs mode for viewing Diffusion Log files
May 01, 2014
Hey – the ’90s called! They said an emacs mode for viewing Diffusion log files would be real handy. I’ve just pushed diffusion-log-mode to GitHub. The mode provides simple syntax highlighting and view-mode navigation for logs produced by Diffusion 4.6 and higher. …
A Maven plugin for creating DAR files
August 13, 2013
Summary Apache Maven has become the de facto build tool for Java projects. The Push development team uses it to create the Diffusion™ Server and sample applications. We have and are continuing to improve our support for creating Diffusion Applications using Maven, allowing our users to benefit from its declarative approach to full-lifecycle builds, component …
Native C/C++ Like Performance For Java Object Serialisation
September 04, 2012
Do you ever wish you could turn a Java object into a stream of bytes as fast as it can be done in a native language like C++? If you use standard Java Serialization you could be disappointed with the performance. Java Serialization was designed for a very different purpose than serialising objects as quickly …
