Wednesday, May 13, 2015

Blueprint: Data Management Challenges with Continuously Streaming Telemetry

by Ravi Mayuram, Senior Vice President of Products and Engineering at Couchbase

As the Internet of Things continues to take shape, millions of connected devices collect and share remote sensor data that organizations use to understand and improve business efficiencies. The process of collecting this remote sensor data – which can be anything from movement to temperature to power status and more – is known as telemetry and it presents a unique challenge for organizations. As a technology, telemetry has been around for decades, but with the rise of the Internet of Things, the characteristics (speed, volume, etc.) of telemetry data have changed significantly. Whether it’s the utility industry leveraging new smart meters to improve the reliability and efficiency of smart grids in real-time, or the airline industry to monitoring all aspects of a jet engine in motion, or blind spot detection in cars, telemetry has become a primary source of Big Data and Big Data requires new capabilities in order to catalog, interpret, and derive value from the data collected.

As a result, most industries that rely on telemetry are implementing new database technologies that have the ability to capture and analyze the vast amounts of incoming real-time data in a millisecond range. These continuous streams of bite-sized data require high-performance databases that can sustain very high throughput, maintain very low latency, and store semi-structured data in a useful and easy-to-understand format.

Challenges of Managing Continuously Streaming Telemetry Data - The Three Vs: Volume, Velocity and Variety 

The biggest challenge for managing streams of telemetry data is being able to keep up with the demands of “Big Data” (as the unprecedented volume of data generated today is called). Big Data has three primary characteristics; its volume (there is an awful lot of it), its velocity (how fast this volume is generated) and its variety (the very different types of data that are generated). All three of these create unique challenges for next-generation databases.

First, consider the performance needed to keep up with the speed with which data is generated in a true Big Data implementation. A single sensor can generate hundreds of readings per second. Your average jet engine may have as many as 250 sensors operating simultaneously. As a result, a single component of a machine can generate hundreds of thousands of readings at the same time. All of the machines within a single oil rig can generate millions, if not tens of millions, of readings per second and a machine cannot wait seconds, or even milliseconds, for a data point to be collected because it is continuously producing new data.

Scalability is a byproduct of the amount of infrastructure required to handle the volume of data generated by a Big Data implementation. It does not take long to accumulate massive volumes of data when generating millions of data points per second. Enterprises now store much, much more data than they used to because the most complete data set enables early detection of trends as well as deeper insights.

As the type or variety of data continues to expand, organizations find they require a flexible data model. For example, when a new sensor is added to a machine or firmware is upgraded (which can be often, depending on the application), the data model changes, as there is new information being captured by this new sensor. Modern NoSQL databases which support flexible schemas take this change in stride and adapt to this new data model in real-time – no upgrade outages. The ability for a database to incorporate a variety of sensor data (new and updated) without having to change application code or relational schemas is what enables these more modern systems to make applications as agile as they are.

NoSQL and Real-Time Database Systems 

As a result of these characteristics for today’s telemetry, more and more organizations are turning to NoSQL databases. These next-generation solutions deliver the performance and scale capable of meeting the challenges of today’s telemetry streams. These systems also deliver significant developer agility, a requisite for today’s telemetry applications. Telemetry data is semi-structured, making it difficult to force into the highly structured format of a relational database. In contrast, instead of rows and columns used in other models, NoSQL data is stored in JSON documents. JSON, or JavaScript Object Notation, is a lightweight, developer-friendly, data-interchange format that is easy to parse and generate. The database is schema-less and thus the data model is flexible, fast to read and write, and easy to scale across commodity servers. This is the ideal format for telemetry data.

The native distributed nature of NoSQL technology is another advantage when massive amounts of data are being stored and accessed. Data is spread across multiple processing nodes and often across multiple servers. As a result, NoSQL databases are referred to as horizontally scalable systems. What this means is that if you find that you need more database capacity you don’t have to buy a more powerful hardware system, you simply add one more server (commodity class hardware will do) to your database server farm, adding additional servers as needed. You can do this based on the immediate need without needing to plan too much in advance and procure sophisticated hardware. This real-time ability to elastically scale this system is a distinct advantage to agile business who cannot predict the data (business) growth a priori.

NoSQL solutions are also ideal for the kind of analysis required by Hadoop-based machine learning, a critical form of advanced analytics that’s capable of identifying deep insights from Big Data sources like telemetry. NoSQL databases provide the best of both worlds when it comes to analytics, since NoSQL enables organizations to reap the benefits of real-time analysis while simultaneously feeding the company’s Hadoop platform the data it needs for deeper, offline analysis.

A high performance, distributed NoSQL database can store raw and/or processed data from stream processors; it can further process the data by sorting, filtering, and aggregating it; and it enables external applications to access raw and processed data via live dashboards for real-time monitoring and visualization. For real-time analysis, sensor data can be ingested as messages for a stream processor to analyze in real-time. The analysis can be written to a NoSQL database for display by live dashboards so enterprises can take immediate action. At the same time the stream processor writes the analysis to the NoSQL database, it continues to write raw sensor data to Hadoop for offline analysis. This allows enterprises to combine real-time analytics and data access with offline analytics to improve both short-term and long-term operations.


Continuously streaming telemetry data has the potential to transform industries from healthcare, to energy, to transportation. But in order to maximize its potential, we need the right Big Data architecture in place. The combination of NoSQL and Hadoop gives organizations a modern data architecture that can continuously analyze streaming data, make smart decisions in milliseconds, and intelligently evolve system behavior to respond to events happening in real-time.

About the Author
Ravi Mayuram is the Senior Vice President of Products and Engineering at Couchbase. He leads product development and delivery of the company’s NoSQL offerings. He was previously with Oracle where he was a senior director of engineering leading innovations in the areas of recommender systems and social graph, search and analytics, and lightweight client frameworks. He was also responsible for kickstarting the cloud collaboration platform. Ravi has also held senior technical and management positions at BEA, Siebel, Informix and HP in addition to couple of startups including BroadBand office, a Kleiner Perkins funded venture. Ravi holds a MS in Mathematics from University of Delhi.

About Couchbase
Couchbase delivers the world’s highest performing NoSQL distributed database platform. Developers around the world use the Couchbase platform to build enterprise web, mobile, and IoT applications that support massive data volumes in real time. All Couchbase products are open source projects. Couchbase customers include industry leaders like AOL, AT&T, Bally’s, Beats Music, BSkyB, Cisco, Comcast, Concur, Disney, eBay, Intuit, KDDI, Nordstrom, Neiman Marcus, Orbitz, PayPal, Rakuten / Viber, Ryanair, Tencent, Verizon, Wells Fargo, Willis Group, as well as hundreds of other household names. Couchbase investors include Accel Partners, Adams Street Partners, Ignition Partners, Mayfield Fund, North Bridge Venture Partners, and West Summit.

Got an idea for a Blueprint column?  We welcome your ideas on next gen network architecture.
See our guidelines.

Advantech: Faster Development with a Deep NFV Ecosystem

Advantech is building a deep ecosystem for accelerating the rollout of NFV infrastructure. In this video, Paul Stevens introduces the Advantech ecosystem, Renu Navale discusses participation in Intel's Network Builders community , Kelly LeBlanc discusses 6Wind's high-performance virtual accelerator technology, and Charlie Ashton presents the Wind River Titanium Server NFV infrastructure solution.

See video:

Cisco Sees Strength in Data Centers and Cloud, SP Business Remains Challenged

Cisco reported third quarter revenue of $12.1 billion, net income on a generally accepted accounting principles (GAAP) basis of $2.4 billion or $0.47 per share, and non-GAAP net income of $2.8 billion or $0.54 per share. Overall revenue was up 5.1% compared to the same period a year ago.

"Cisco is in a very strong position and we delivered another solid quarter. Our vision and strategy are working and we are executing very well in a tough environment, as evidenced in our revenue growth, profitability, strong gross margins and cash generation. Our customers feel the pace of change and disruption in every industry and market, and know their success depends on digitizing their business. Whether they are the disruptor or the incumbent, they are coming to Cisco as their strategic partner. We believe we are pulling away from our competition using the same formula we've always used: integrating our industry-leading products in every category into architectures and solutions that deliver real outcomes. We've created this opportunity and it is ours to execute," stated John Chambers, Cisco chairman and CEO.

"I am extremely honored and proud to have led Cisco for the last 20 years and to get us to this positive inflection point. We have a tremendous opportunity to extend our lead in the industry, and with Chuck Robbins as the CEO for Cisco's next chapter, we have exactly the right leader to capture that opportunity. I could not be more confident in our future."

Some highlights:

  • U.S. Public sector growth of 10%
  • Service Provider business remains "challenged". Orders declined 7% globally and 17% in the U.S.
  • Nexus 3K and 9K sales were up 144% y/y. Cisco now has 2,650 Nexus 9K and ACI customers, as well as over 580 APIC controller customers
  • Nexus 9K + APIC orders were up 27% q/q, and Cisco expects growth to continue in Q4
  • Data center revenues were up 21% y/y with UCS now on an over $3 billion run rate and over 43,800 customers
  • NGN Routing was up 4% y/y
  • Gross margins remain solid in switching
  • Wireless was up 9% y/y/ and Meraki was up 92% y/y
  • Security was up 14% y/y
  • Collaboration was up 7% y/y
  • There were over 15 million users of Webex
  • Returned $2.1 billion to shareholders through share repurchases and dividends

Citrix Workspace Cloud Aims for On-Demand IT

Citrix is launching a new cloud platform designed to simplify the on-demand delivery of a wide range of IT tools and services.

The Citrix Workspace Cloud creates a new control plane that merges the on-premises and cloud, allowing IT managers to create secure, mobile workspaces that include desktops, applications and data from whatever infrastructure source best meets their specific needs.

Citrix said its cloud-based management platform provides customers the flexibility to choose which clouds or datacenter resources fit their needs best, then select the applications, data, files and features that work best for each specific organization and each person. Partners add value by architecting and designing customized, personalized workspaces uniquely suited to meet their customers’ business requirements.

“Citrix Workspace Cloud is the future of on-demand IT. People want access to all their apps and data, and this no longer equates to a desktop. Citrix has created the fastest and easiest way to deploy new resources, simplified infrastructure management, and provided freedom of choice in selecting the right hosting and delivery model. Today, we are excited to offer this test drive to our partners and customers so they can see how Citrix Workspace Cloud supports their business and on-demand IT strategies.”

CenturyLink Opens Hydro-electric-powered Data Center

CenturyLink inaugurated a new data center in central Washington, adding hydro-electric-powered data center services to its hybrid IT services portfolio.

The data center, located in Moses Lake, Washington, will ultimately support up to 30 megawatts of IT load on the site, with an initial ramp to 8 megawatts. It offers data center and network services, and includes access to cloud, colocation and managed services available through CenturyLink's global data center footprint. The site also has a seismic rating of 2B, the lowest rating in the western United States, making it a key location for disaster recovery solutions.

"CenturyLink's new low-cost power data center services provide many benefits to our customers, including a highly resilient solution coupled with power costs and efficiency metrics that rank among the best in the industry, and the facility serves as an excellent disaster recovery location," said David Meredith, senior vice president, CenturyLink. "Enterprises enjoy global access to CenturyLink's portfolio of cloud and managed hybrid IT services, and we continue to extend the reach of our data center footprint to new markets to meet from the needs of our customers."

The property is owned by Server Farm Realty and leased to CenturyLink.

VeloCloud Exceeds ONUG Testing for Software-Defined WANs

VeloCloud Networks, a cloud-delivered SD-WAN (software-defined wide area network) company, has exceeded all Open Networking User Group (ONUG) SD-WAN tests. ONUG is a user-driven community of IT executives focused on open networking solutions. Its board includes IT leaders from Bank of America, Cigna, Citigroup, Credit Suisse, FedEx, Fidelity Investments, Gap Inc., JPMorgan Chase, the Lippis Report, Morgan Stanley, Pfizer, Symantec and UBS. An ONUG meeting is underway this week in New York.

VeloCloud said the extensive ONUG testing process, which lasted three months, aim to provide proof of concepts, feature validation and demonstrations to ensure that the networking solutions in the network overlays, SD-WAN and network services virtualization areas deliver on the top 10 requirements developed by the ONUG Working Groups. The testing was conducted by Ixia, in conjunction with ONUG.

“Our success in ONUG’s landmark Top Ten requirements testing for SD-WAN is an important milestone demonstrating VeloCloud’s SD-WAN solution alignment with the needs of enterprises,” said Sanjay Uppal, CEO and co-founder of VeloCloud. “As the Cloud-Delivered SD-WAN leader, we are committed to working closely with ONUG and the IT community to accelerate the adoption of secure and innovative technologies that dramatically improve the economics, performance and security of enterprise applications.”

“These tests underscore our commitment to the ONUG community of IT executives and vendors to validate product features that meet IT business leader requirements for enterprise cloud infrastructure,” said Nick Lippis, chairman and co-founder of ONUG. “We are confident that this testing effort with companies such as VeloCloud will assist in moving the open infrastructure industry from technology curiosity to operations.”

Alcatel-Lucent Extends Reach of Unrepeatered 100G Subsea Links

Alcatel-Lucent Submarine Networks has extended the reach of unrepeatered cable systems at 100 Gbps to more than 610 kilometers using the same fiber for both signal and amplifier transmission.

The company said this advancement will lead to significant improvements to cable system efficiency, creating benefits for operators in terms of total cost of network ownership. Mainly driven by connectivity needs, unrepeatered undersea cable projects consist in the construction of cable systems for distances currently up to 500km without using repeaters to amplify the signal. Built to offer additional capacity and/or complement terrestrial networks to gain resiliency, they can offer a design capacity of 25 Tbit/s depending on the system configuration.

Olivier Gautheron, Chief Technology Officer of Alcatel-Lucent Submarine Networks said: “Operators’ infrastructures continue to be challenged by the demands of traffic arising from  broadband services and content delivery. ASN’s breakthrough further demonstrates the application of our own innovation to extend the reach of unrepeatered submarine cable systems in order to enhance their performance to help operators get the most out of their networks and build new revenue streams.”

ASN has so far deployed more than 20,000 km of unrepeatered cable systems in more than 40 projects for major telecom operators worldwide.

Clouditalia Deploys Coriant for Packet/Optical

Clouditalia, a provider of integrated telecommunications and cloud services throughout Italy, has deployed the Coriant 7090 Packet Transport Platform to enhance service delivery to its Small and Medium Enterprise (SME) business customers.

The Coriant Packet Transport Network (PTN) solution, which includes support for MPLS-TP, enables Clouditalia to cost-effectively deliver flexible Gigabit Ethernet (GbE) services directly to end-user business customers, while expanding support for growing cloud and Data Center Interconnect (DCI) applications. With the ability to efficiently support diverse traffic types on the same platform, the Coriant solution also enables Clouditalia to enhance customer value by seamlessly migrating legacy services such as E1, SDH, and lower bit rate Ethernet.