Site Reliability Engineer

github

Raphael Ribeiro  • Peerlist

Currently, there is a great demand for computing services provided by cloud computing.  For example, large-scale scientific experiments involving parallel and computationally intensive executions utilize cloud computing infrastructures as the execution environment. However, it is still not a trivial task to configure, deploy and manage these infrastructures. On this context, we present a simplified solution for executing a distributed cloud computing environment based on the master/workers paradigm.  The proposed system, entitled Infra.jl, creates, manages and monitors workers from Julia language in the cloud for distributed data processing. Experiments with Infra.jl show that, in addition to facilitating the construction of hundreds of workers, this task is done automatically in minutes. In addition, Infra.jl allows scientists to improve their productivity through automatic resources provisioning. Infra.jl is free software and is available at https://github.com/gsd-ufal/Infra.jl together with the support material that allows the reproduction of the results presented in this work.


Infra.jl, a tool for managing Julia workers as containers in

Current cloud programming support for highly-responsive machine learning applications are either complex to use or provide limited features. This research project aims at developing the CloudArray support for highly-responsive machine learning applications. CloudArray provides high-level cloud programming support which transparently deals with resource management based on service-level objectives. Expected results include (i) open-source and standard cloud programming support for building highly-responsive machine learning applications; (ii) two Azure-based open-source applications able to solve real-world problems; (iii) further contributions to the open-source community such as training material and pull requests.

CloudArray: highly-responsive machine learning in the cloud 

Infra.jl books virtual machines (VMs) and creates, configures, and instantiates Docker containers on top of VMs. Then Julia Workers are configured and deployed on containers. 

Infra.jl: A Julia interface to launch workers through Micros

The development of technologies for high throughput data acquisition in a number of domains (e.g. biological sciences, environmental sciences, astronomy, meteorological sciences, etc.), together with advances in digital storage, computing, and communications technologies have resulted in unprecedented opportunities for scientist to use the wealth of distributed information available in learning, scientific discovery, and decision making. Classic Data Mining/Machine Learning architectures do not fulfill such needs, due to high communication, storage and computing costs (among other problems). Due to the huge amount of data distributed in different sources, it is infeasible to collect all this information and store them in a centralized data source. That is why it is necessary to propose novel approaches for classic Machine Learning algorithms to work in Distributed Environments. We call this Adaptive Distributed Machine Learning. There have been many advances in the supervised learning task for horizontally and vertically fragmented scenarios in the last years (separately), but the problem of Distributed Machine Learning with hybrid data fragmentation (both scenarios at the same time) has not been fully addressed. Also the problem where batches of data examples arrive over time, has not been considered in the state of the art of the Distributed Machine Learning community. In non-stationary environments, these batches of data may suffer from concept drift, meaning that the underlying law of probability that generates the data may change over time. Also, if we add the inherent complexity of handling very large data sets that are distributed, the full task becomes very challenging. This project deals with Distributed Learning problems, with different underlying laws of probability governing the distributed sources, in incremental scenarios, minimizing the communication, storage and processing costs.