CloudEXPO 2019

Addressing Critical Shortcomings of ETL Tools for Better Data Analytics

Poor data quality and analytics drive down business value. In fact, Gartner estimated that the average financial impact of poor data quality on organizations is $9.7 million per year. But bad data is much more than a cost center. By eroding trust in information, analytics and the business decisions based on these, it is a serious impediment to digital transformation.

Extract, transform and load (ETL) tools like AWC Glue bring much needed functionality. This tool enables new approaches to pulling, processing and pushing data from source to target, and introduces concepts such as performing data transformation tasks using SparkSQL scripts in Apache spark environment. However, there are shortcomings with AWS Glue, leading to a number of challenges and questions:

Where are the necessary coding techniques when it comes to dealing with specific data types and many such technical aspects of traditional ETL tools?

How can data analysts rely on the Glue-Dynamic frame concept for some of the key design aspects when it comes to incremental loads design and data conversion process?

How can accurate reporting be achieved when the data processing step truncates decimals points, creating huge data discrepancies?

At Infostretch, we believe these challenges can be handled by utilizing specific database level functions. In this session, we will showcase real-life experience with such deal-breaking scenarios, and demonstrate how to mitigate these issues without jeopardizing reporting accuracy, compromising on quality, or endlessly waiting for new releases.

Attendees will learn how to overcome the critical shortcomings of the AWS Glue ETL tool to achieve success:

How to use specific database level functions without jeopardizing quality

Address issue of truncating decimals to produce accurate reports

Ensure source system KPIs match up with target system KPIs for complete business insights.

Serverless Made Media Stream Processing Simpler But Powerful

Function as a Service is the popular solution of Serverless, which runs a short-lived function. In contrast, we will show a new Serverless: run long-lived function to deal with stream data. Running containers per stream, each container's code footprint became small, and the system complexity got significantly simple. This made it easy to realize a new use case like enriching an application with AI-powered analytics using video stream in real-time.

NTT Communications has developed a new platform with Kubernetes to orchestrate this new style of media stream processing. We will throw a live demo to send voice stream from browser to this platform and run media processing functions in real-time. The functions will be open sourced, and all participants can try it out from tomorrow.

Castle Walls Won’t Protect Your Public Cloud

The public cloud is not inherently insecure, yet the news is full of stories on public cloud data breaches. Many organizations fail to secure their public cloud resources because they attempt to apply the same security models that they used in their own data center. Join BMC to learn about the new threat vectors and the best ways you can protect your public cloud resources.

Know More About Your Ceph Cluster with ELK Stack

Ceph generates quite a few logs which you can use to transform into alerts and do proper analysis of the data. If you tune things right you just might catch a problem before it occurs. Maybe you did have a problem already and you want to analyze it so that it doesn't happen again. We will explore how to enable GELF logging in Ceph and utilizing that with the ELK Stack. Another use we can tie into the ELK stack is the Object storage (RGW). With the use of elasic search we can now search our metadata and sort through important information being stored as objects. This metadata can also be customized with your own fields in RGW. I'll demonstrate live some analysis of a running Ceph cluster with both the logging and RGW metadata search abilities.

UniModal IT for Financial Services

Back in 2014 Gartner came up with the term “Bi-Modal” – with Mode 1 referring to the legacy systems or the System of Records, which was seen more as “keeping the lights on” mode, much to do with stability, whereas Mode 2 referred to the System of Innovation or the System of Differentiation with the need to deal with speed, innovation, and agility.
While there is varied opinion around the very notion of Bi-modal IT, this term has been sold eloquently to C-Suite and for the last 3-4 years CIO’s have been grappling with - “how to deal with 2 speed IT”- essentially how to transform while keeping the light on.
This talk essentially will provide the audience with the insight of "how to move to a Single Speed or Unimodal IT, where they can think of achieving IT Agility not only in the System of Engagement but also in the System of Records. Using DevOps as the underlying lever this talk will elaborate on how to gain speed and velocity without comprising on stability and resiliency across the application portfolio. It will explain how to stand up an end to end integrated DevOps pipeline not only on the distributed technologies (java, .net, node) but also on the legacy side of the house including Mainframe, iseries, and AS400. It will further introduce some of the best practices which the enterprise as a whole needs to adopt to graduate to a Unimodal IT.

The Hybrid Cloud Chess Match

As enterprise workloads proliferate, the dynamic nature of hybrid cloud deployments make the management of both cyber threats and compliance a significant challenge.According to RightScale, on average, companies are using nearly five different cloud environments at a time. Just over one-half are deploying a hybrid cloud strategy, as opposed to only one of five which are committed to a multiple public cloud environment and one of ten which are deploying a multiple private cloud strategy.

In this talk, Caveonix CEO and co-founder, Kaus Phaltankar will address how enterprises must manage their risk posture given this new state of complex hybrid cloud environments, regardless of where workloads reside.

To win in chess, you must stay at least one step ahead of your opponent by anticipating what you have that’s valuable and then developing strategies to keep it “safe.” Workload protection functions very much in the same manner, only at a far greater scale.

Hybrid cloud workloads are highly volatile and have a multitude of potential configuration issues, and potential vulnerabilities due to their installed software versions and patches. They may also have dynamic inter-dependencies and exposure to other application workloads.

In multi-cloud environments, each individual cloud may be based on different technology implementations that result in inconsistent security and policy frameworks. This makes the management of cybersecurity and compliance requirements across the hybrid cloud increasingly complex.

Kaus will address how to protect these workloads in a hybrid cloud environment. Attendees will learn why enterprises need to have a consistent security solution that understands what is being deployed, where it is deployed, what has changed, how they may be subject to the latest threats, how to deploy consistent security policies, and how to proactively mitigate these risks.

Get Behind Your OpenStack and Transform Your Adoption

When it comes to OpenStack adoption you might need to think of some more ways in which you can successfully adopt it in your organization. As I have been operating OpenStack for several years there have been some key objectives which will help increase your adoption.

I'll go over a few of these objectives from workload migration including P2V,V2V,C2C,H2H to Workload transformation. I will talk about the process, requirements, tools, and leave you with all you need to press forward with greater OpenStack adoption.

I will provide a demonstration of some of the tools in action so you can see how you can successfully inspect a workload, prepare it, and build it for running on OpenStack all in an automated fashion with little to few steps required. The goal is to leave you with the confidence that this has been done before and you can accomplish it too.

Establish a Successful and Sustainable IaaS Business

Infrastructure as a service (IaaS) is the fastest-growing segment of the public cloud services market, which Gartner has forecasted to hit an estimated $57.8B in 2019. The speed at which IaaS is growing presents a unique opportunity for channel resellers through Ingram Micro Cloud. Learn what’s required for building a successful IaaS practice, such as identifying solutions and platforms, structuring talent, choosing value-added tools and resources, creating a services strategy, and more.

Bringing the Software Defined Data Center into Focus for SME Customers

The VMware NSX platform represents a paradigm shift for small-to-medium enterprises (SMEs). By leveraging multi-tenant cloud environments that leans on the NSX platform, SMEs can gain the network virtualization and security benefits of a software-defined data center (SDDC), that were previously out of reach, except for large corporate organizations.

Attend This Educational Session to Hear Such Topics As:

• Defining the software-defined data center (SDDC) approach and what it should mean to SMEs.
• Examining time and resource improvements gained by SDDC virtualization and security.
• Realizing security at a virtual machine (VM) level through “micro-segmentation” process. 
• Simplifying backup and disaster recovery services.
• Learning how managed services can help remove any hesitation of an SSDC approach.

Public Cloud Performance: Which Is the Right Cloud Provider for Your Business?

Given the proliferation of the public cloud within the enterprise architecture and the dominance of the Big 3 – AWS, Google Cloud Platform (GPC), and Microsoft Azure – this session will present an agnostic, first-of-its-kind comparison of the performance of the three major public cloud vendors.

Ultimately, it is important for IT leaders to understand that cloud architectures are complex and not to be taken lightly. Enterprises heavily relying on the public cloud, spanning multiple AZs, regions, and providers should arm themselves with the right data to understand performance and changes over time.
Overall, the collected performance data indicates comparable and robust network performance across all three cloud providers, yet geographical exceptions persist. Cloud connectivity architectures vary between the three providers, with corresponding differences in the level of Internet exposure when connecting to the cloud.

Given the globally distributed nature of customers, their client base, and infrastructure, plus the inherently variable nature of the Internet, monitoring global performance of cloud providers is imperative for determining how their application or service performs on a global scale.

Relying on the cloud means relying on the Internet – that black box shrouded in a cloak of invisibility that no enterprise owns or controls – yet businesses rely on it for outcomes. And as multi-cloud increasingly becomes the norm, visibility issues are significantly compounded. Whether an enterprise is deploying G Suite and wants to monitor performance across branch offices, or if it wants to deploy its own applications hosted on GCP and ensure it has the most control possible over the end-user experience, visibility into how these applications are traversing the Internet across multiple service providers is critical for ensuring exceptional digital experiences

Large Affordable Server Memory for Public and Private Cloud

DRAM pricing remains the highest component cost in server economy. Even in public cloud, large memory instances cost in multiples more than smaller-memory systems. Further, DRAM scale is quite limited, to the 6TB range in on-premise deployments and even lower in cloud, which makes some workload impossible to run.

ScaleMP's MemoryONE is solving these problems with Software-defined Memory leveraging affordable NVM as system memory, transparently. Using MemoryONE, you can significantly reduce your cost by expanding memory beyond specification limits - to reduce the total number of servers for your application, or by replacing majority of the DRAM expense with NVM and software-defined Memory. MemoryONE is a drop-in replacement, with no change to the rest of the hardware and software stack, and the performance of MemoryONE is near-DRAM as shown by many benchmarks for cloud applications.

In this technical presentation, we will walk through the basics of Software-defined Memory, how it is implemented in MemoryONE, and also do a hands-on demo.

Chasing Your Startup Dream? Let's Talk About How We Can Make It Happen!

I know we're all here to talk tech and learn new cool stuff, but I'm sure many of you here have at least five ideas running through their heads, and they wish they can one day make it happen and start their startup.

Now I know what's running in your head now. You have a fat paycheck every month, health insurance, family commitments, monthly bills, fear of opportunity loss, etc. And I bet you always talk to some of your friends about your excellent idea that will rock the world as soon as you launch it BUT do you always say "I want to have all my ducks in a row before I quit my lucrative job and start my company."?
Guess what, your ducks will not be in a row, your plan will never be crystal clear, you have to take some risks and go through some rough conditions and many unknowns.

In this session let's talk about some of the lessons learned to start up a business. How to calculate your risks? And is it worth it?

Feel free to email me questions in advance to omar.mehilba@magalix.com, and I will cover in my talk.

Building Maintainable, Observable Applications on Serverless Architecture

Using serverless computing has a number of obvious benefits over traditional application infrastructure - you pay only for what you use, scale up or down immediately to match supply with demand, and avoid operating any server infrastructure at all.

However, implementing maintainable and scalable applications using serverless computing services like AWS Lambda poses a number of challenges. The absence of long-lived, user-managed servers means that states cannot be maintained by the service. Longer function invocation times (referred to as cold starts) become very important to track, because they impact the response time of the service and will impose additional cost. Additionally, the transition to smaller individual components (much like breaking a monolithic application into microservices) results in a simpler deployment model, but makes the system as a whole increasingly complex.

In this talk, the speaker will discuss patterns and best practices around architecting and implementing code in serverless environments, specifically around how to build maintainable serverless code and minimize the occurrence of cold starts. Additionally, the speaker will cover how to properly instrument applications and supporting services so that your systems remain easily observable.

Dynamic Auto Scalable Software Applications

Dynamic auto scalable software applications having multiple custom schedulers running as part of a Multi-Cluster often have the need to execute code in exactly one Application Instance. This pattern can be used to implement processing which must only be performed once within the entire application. This is achieved via a leader election and even job distribution. Microservices utilize this software pattern to ensure all future schedulers that will wake up in the future will stream the events for marketing campaigns. It is so important that we should not reach a customer with several duplicate messages through multiple channels like email, Direct Mail etc. When a custom configured, persisted scheduler wakes only one Application Instance will start processing the customer information to determine the eligibility to reach a customer and stream those events through the specific channels. Ensuring exactly one Node/Scheduler election to run a custom scheduled job ensures events are not duplicated resulting in duplication of work and wasted computer resources from technology and duplicate campaigns from business perspective.

Also, a Scheduler is non-preemptive. Thus, if an Application Instance is running the Scheduler and another Application Instance is eligible to run the Scheduler, the eligible Application Instance must not preempt the Scheduler already running; running Scheduler is allowed to complete.
Apart from that having multiple custom configured scheduled jobs, this algorithm should be responsible to evenly and randomly choosing a node for load distribution of the job election leaders.
The leader election algorithm is aware of the Application Instance version. If a Multi- Cluster contains Application Instances with different versions the Scheduler must only run in the newer version Application Instances.