Wednesday, January 17, 2018

Deep Complex Networks - implementation -

Here is some interesting work on complex DNNs. 




At present, the vast majority of building blocks, techniques, and architectures for deep learning are based on real-valued operations and representations. However, recent work on recurrent neural networks and older fundamental theoretical analysis suggests that complex numbers could have a richer representational capacity and could also facilitate noise-robust memory retrieval mechanisms. Despite their attractive properties and potential for opening up entirely new neural architectures, complex-valued deep neural networks have been marginalized due to the absence of the building blocks required to design such models. In this work, we provide the key atomic components for complex-valued deep neural networks and apply them to convolutional feed-forward networks and convolutional LSTMs. More precisely, we rely on complex convolutions and present algorithms for complex batch-normalization, complex weight initialization strategies for complex-valued neural nets and we use them in experiments with end-to-end training schemes. We demonstrate that such complex-valued models are competitive with their real-valued counterparts. We test deep complex models on several computer vision tasks, on music transcription using the MusicNet dataset and on Speech Spectrum Prediction using the TIMIT dataset. We achieve state-of-the-art performance on these audio-related tasks.
Implementation is here: https://github.com/ChihebTrabelsi/deep_complex_networks

The reviews are here. Also, Carlos Perez had a blog post on the matter a while ago: Should Deep Learning use Complex Numbers?




Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Tuesday, January 16, 2018

Understanding Deep Representations through Random Weights




When it comes to random projections and Deep Neural Networks, the paper following paper is intriguing:
In summary, applying random weights in the whole CNNDeCNN architecture, we can still capture the geometric positions and contours of the image. The shape reduction of feature maps takes responsibility for the randomness on the reconstructed images for higher layer representation due to the representation compression. And random weight DeCNN can reconstruct robust images if we have enough number of feature maps



We systematically study the deep representation of random weight CNN (convolutional neural network) using the DeCNN (deconvolutional neural network) architecture. We first fix the weights of an untrained CNN, and for each layer of its feature representation, we train a corresponding DeCNN to reconstruct the input image. As compared with the pre-trained CNN, the DeCNN trained on a random weight CNN can reconstruct images more quickly and accurately, no matter which type of random distribution for the CNN's weights. It reveals that every layer of the random CNN can retain photographically accurate information about the image. We then let the DeCNN be untrained, i.e. the overall CNN-DeCNN architecture uses only random weights. Strikingly, we can reconstruct all position information of the image for low layer representations but the colors change. For high layer representations, we can still capture the rough contours of the image. We also change the number of feature maps and the shape of the feature maps and gain more insight on the random function of the CNN-DeCNN structure. Our work reveals that the purely random CNN-DeCNN architecture substantially contributes to the geometric and photometric invariance due to the intrinsic symmetry and invertible structure, but it discards the colormetric information due to the random projection.




Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Monday, January 15, 2018

The stochastic interpolation method. The layered Structure of Tensor Estimation and Phase Transitions, Optimal Errors and Optimality of Message-Passing in GLMs



Jean just sent me the following:
Dear Igor,

We have of bunch of recent rigorous results that might be of interest for the community. In order to obtain them, I developed together with Nicolas Macris a new adaptive interpolation method well designed for treating high-dimensironl Bayesian inference problems.

_In this article we present our method with application to random linear estimation/compressive sensing as well as to symmetric low-rank matrix factorization and tensor factorization.

https://arxiv.org/pdf/1705.02780.pdf

_In this one, presented at allerton this year, we present a nice application of the method to the non-symmetric tensor factorization problem that was resisting until now. Moreover we exploit the structure in « layers » of the model, which might be an idea of independent interest.

https://arxiv.org/pdf/1709.10368.pdf

_Finally, our main recent result is the application of the method for proving the statistical physics conjecture for the single-letter « replica formula » for the mutual information of generalized linear models. There we also rigorously derive the inference and generalization errors of a large class of single layer neural networks such as the perceptron.

https://arxiv.org/pdf/1708.03395.pdf

All the best

Thanks Jean, here are the papers you mentioned. I had mentioned one before and I like the layer approach of the second paper !




In recent years important progress has been achieved towards proving the validity of the replica predictions for the (asymptotic) mutual information (or "free energy") in Bayesian inference problems. The proof techniques that have emerged appear to be quite general, despite they have been worked out on a case-by-case basis. Unfortunately, a common point between all these schemes is their relatively high level of technicality. We present a new proof scheme that is quite straightforward with respect to the previous ones. We call it the stochastic interpolation method because it can be seen as an extension of the interpolation method developped by Guerra and Toninelli in the context of spin glasses, with a trial "parameter" which becomes a stochastic process. In order to illustrate our method we show how to prove the replica formula for three non-trivial inference problems. The first one is symmetric rank-one matrix estimation (or factorisation), which is the simplest problem considered here and the one for which the method is presented in full details. Then we generalize to symmetric tensor estimation and random linear estimation. In addition, we show how to prove a tight lower bound for the mutual information of non-symmetric tensor estimation. We believe that the present method has a much wider range of applicability and also sheds new insights on the reasons for the validity of replica formulas in Bayesian inference.

We consider rank-one non-symmetric tensor esti- mation and derive simple formulas for the mutual information. We start by the order 2 problem, namely matrix factorization. We treat it completely in a simpler fashion than previous proofs using a new type of interpolation method developed in [1]. We then show how to harness the structure in "layers" of tensor estimation in order to obtain a formula for the mutual information for the order 3 problem from the knowledge of the formula for the order 2 problem, still using the same kind of interpolation. Our proof technique straightforwardly general- izes and allows to rigorously obtain the mutual information at any order in a recursive way.



We consider generalized linear models (GLMs) where an unknown n-dimensional signal vector is observed through the application of a random matrix and a non-linear (possibly probabilistic) componentwise output function. We consider the models in the high-dimensional limit, where the observation consists of m points, and m/n→α where α stays finite in the limit m,n→∞. This situation is ubiquitous in applications ranging from supervised machine learning to signal processing. A substantial amount of theoretical work analyzed the model-case when the observation matrix has i.i.d. elements and the components of the ground-truth signal are taken independently from some known distribution. While statistical physics provided number of explicit conjectures for special cases of this model, results existing for non-linear output functions were so far non-rigorous. At the same time GLMs with non-linear output functions are used as a basic building block of powerful multilayer feedforward neural networks. Therefore rigorously establishing the formulas conjectured for the mutual information is a key open problem that we solve in this paper. We also provide an explicit asymptotic formula for the optimal generalization error, and confirm the prediction of phase transitions in GLMs. Analyzing the resulting formulas for several non-linear output functions, including the rectified linear unit or modulus functions, we obtain quantitative descriptions of information-theoretic limitations of high-dimensional inference. Our proof technique relies on a new version of the interpolation method with an adaptive interpolation path and is of independent interest. Furthermore we show that a polynomial-time algorithm referred to as generalized approximate message-passing reaches the optimal generalization error for a large set of parameters.









Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Friday, January 12, 2018

OpenMined Hackathon in Paris (Saturday, January 13th)



I heard about the OpenMined project during the Paris Machine Learning meetup that we organized back in December from a presentation by Morten DahlOpenMined is a community focused on building open-source technology for the decentralized ownership of data and intelligence.


"It is commonly believed that individuals must provide a copy of their personal information in order for AI to train or predict over it. This belief creates a tension between developers and consumers. Developers want the ability to create innovative products and services, while consumers want to avoid sending developers a copy of their data.

With OpenMined, AI can be trained on data that it never has access to.
The mission of the OpenMined community is to make privacy-preserving deep learning technology accessible to consumers, who supply data, and machine learning practitioners, who train models on that data. Given recent developments in cryptography, AI-based products and services do not need a copy of a dataset in order to create value from it."


The community is organizing a hackathon 

HACKATHON
On Saturday, January 13th, the OpenMined community will be gathering in-person in over 20 cities around the world to collaborate on various coding projects and challenges. We’ll have a worldwide video hangout for all who cannot make it to a physical location. The hackathon will include three coding projects, each with a live tutorial from a member of the OpenMined community.
Here are the general details:

OpenMined Hackathon DetailsDate: January 13, 2018

On Saturday, January 13th, the OpenMined community will be gathering in-person in over 20 cities around the world to collaborate on various coding projects and challenges. We’ll have a world-wide video hangout for all who cannot make it to a physical location. The hackathon will include three coding projects, each with a live tutorial from a member of the OpenMined community. While hackathons will start at the discretion of each city’s organizer (slack them for details), code tutorials will be live broadcasted at 3 different times: 12:00 noon London time, 12:00 noon Eastern time, and 12:00 noon Pacific Time.



Coding Projects
Beginner: Build a Neural Network in OpenMinedPresentation: How to use the OpenMined Keras InterfaceProject: Find a new dataset and train a new neural network using the Keras interface!
Intermediate: Building the Guts of a Deep Learning FrameworkPresentation: How OpenMined Tensors Work - The Magic Under the HoodProject: Add a feature to Float Tensors
Advanced: Performance Improvements - GPUs and NetworkingPresentation: Optimizing the key bottlenecks of the systemProject: The Need for Speed - Picking a Neural Network and Making it Faster



Physical Locations
Participants in this hackathon will meet in person at the following locations.  If your city says “venue tbd”, reach out to the Slack Point of Contact for specific details and directions. Starbucks is the suggested backup venue of choice - usually has fast wifi and big tables available. (If you aren’t on our Slack, click here for an invite  
Before you come...
you need to do the following 
Join our Slack  
- and join the #hackathon channel 
Reach out to your city’s organizer on Slack! 
Download and Install Unity  
- https://unity3d.com/ 
Follow the Readme to setup OpenMined and PySyft available here: https://github.com/OpenMined/PySyft & https://github.com/OpenMined/OpenMined



OpenMined is a community focused on building technology for the decentralized ownership of data and intelligence.Join our Slack channel to get involved at https://openmined.org/Follow: https://twitter.com/openminedorgContribute: https://github.com/OpenMined/OpenMined







In the OpenMined Slack, there is a #paris channel, the hackaton will be hosted at La Paillasse thanks to support from LightOn. You can find all the details in the #paris channel on the OpenMined Slack.




Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !

Random Incident Sound Waves for Fast Compressed Pulse-Echo Ultrasound Imaging


Martin just sent me the following

Dear Igor, 
I recently discovered your post about the paper "Compressive 3D ultrasound imaging using a single sensor" (http://advances.sciencemag.org/content/3/12/e1701423) by Kruizinga et al. (https://nuit-blanche.blogspot.de/2017/12/compressive-3d-ultrasound-imaging-using.html) and read it with great interest. Thank you very much for highlighting this important contribution! I have been working on the incorporation of CS into ultrasound imaging for several years now (https://scholar.google.de/citations?user=X35rUbAAAAAJ&hl=de) and independently discovered a very similar method for high-frame rate ultrasound imaging (UI). Instead of minimizing the number of sensors, this method aims at minimizing the number of sequential pulse-echo measurements per image. It emits three types of random ultrasonic waves to reduce the coherence of the sound waves scattered by distinct basis functions, e.g. point-like scatterers or Fourier basis functions. The synthesis of these waves exploits the degrees of freedom provided by modern UI systems combined with planar transducer arrays. Specifically, it leverages random time delays, random apodization weights, and combinations thereof. (In essence, the method electronically realizes the fixed coding mask used by Kruizinga et al. as one type of random incident ultrasonic wave.) arXiv.org provides a preprint of this work: https://arxiv.org/abs/1801.00205
I hope that my method appeals to you and the readers of your blog. It would also mean a lot to me, if you mentioned this work on occasion.
Happy new year and keep up your good work


Thank you Martin ! Here is the paper:



A novel method for the fast acquisition and the recovery of ultrasound images disrupts the tradeoff between the image acquisition rate and the image quality. It recovers the spatial compressibility fluctuations in weakly-scattering soft tissue structures from only a few sequential pulse-echo measurements of the scattered sound field. The underlying linear inverse scattering problem uses a realistic d-dimensional physical model for the pulse-echo measurement process, accounting for diffraction, the combined effects of power-law absorption and dispersion, and the specifications of a planar transducer array. Postulating the existence of a nearly-sparse representation of the spatial compressibility fluctuations in a suitable orthonormal basis, the compressed sensing framework ensures its stable recovery by a sparsity-promoting ℓq-minimization method, if the pulse-echo measurements of the individual basis functions are sufficiently incoherent. The novel method meets this condition by leveraging the degrees of freedom in the syntheses of the incident ultrasonic waves. It emits three types of random ultrasonic waves that outperform the widely-used steered quasi-plane waves (QPWs). Their synthesis applies random time delays, apodization weights, or combinations thereof to the voltage signals exciting the individual elements of the planar transducer array.


Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !

Thursday, January 11, 2018

CSjob: One PhD and one Postdoc Position, Low-Rank Matrix Approximations

Nicolas just let me know of this opportunity:

PhD/Postdoc Positions, Low-Rank Matrix Approximations

The research group of Nicolas Gillis at the University of Mons (Belgium) is seeking highly qualified candidates for one doctoral and one postdoctoral position funded by the ERC (European Research Council).
This project, entitled Constrained Low-Rank Matrix Approximations - Theoretical and Algorithmic Developments for Practitioners, focuses on the theory, algorithms and applications of low-dimensional representations of high-dimensional data based on low-rank matrix approximations; see https://sites.google.com/site/nicolasgillis/coloramap/overview for a description of the project and the objectives. The successful candidate will have the flexibility to choose a topic within the range of this project.
Your profile is a MSc/PhD in applied mathematics, mathematics, computer science, or a related discipline. You must have an expertise in at least one of these fields: optimization, computational complexity, numerical linear algebra, data mining and machine learning, hyperspectral imaging. You should have excellent programming skills in a numerical language (such as Matlab), and good communications skills, both written and oral, in English.
The positions remain open until they are filled. The starting date is flexible (ideally in September 2018).
Please see https://sites.google.com/site/nicolasgillis/coloramap/ for more details.





Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !

Wednesday, January 10, 2018

Ce soir: Paris Machine Learning Meetup #5 Season 5: IronCar, Mobile AI, Rules mining


The Paris Machine Learning Meetup #5 Season 5: will take place tonight, thank you to Fortia for hosting this meetup and sponsoring the food and drinks afterwards. Streaming is here.



Schedule:
6:30 PM doors opening ; 6:45 PM : talks begin ; 9:00 PM : talks end, 10:00 PM : end

In this short presentation, we will focus on a very practical NLP problem: how to spot automatically investment rules in a Fund Investment prospectus. At the begining it looks like a very out-of-the-box machine learning problem, but we are fast confronted with real world language modelling problems and its complexities. The goal of the talk is to present the (very specific) problem, its difficulties and how we overcame them.
Mobile cellular networks are becoming increasingly complex to manage while classical deployment/optimization techniques are cost-ineffective and thus seen as stopgaps. This is all the more difficult considering the extreme constraints of 5G networks in terms of data rate (more than 10 Gb/s), massive connectivity (more than 1000000 devices per km2), latency (under 1ms) and energy efficiency (a reduction by a factor of 100 with respect to 4G network).
Unfortunately, the development of adequate solutions is severely limited by the scarcity of the actual ressources (energy, bandwidth and space). Recently, the community has turned to a new ressource known as Artificial Intelligence at all layers of the network to exploit the increasing computing power afforded by the improvement in Moore’s law in combination with the availability of huge data in 5G networks.
This is an important paradigm shift which considers the increasing data flood/huge number of nodes as an opportunity rather than a curse. In this talk, we will discuss through various examples how the recent advances in big data algorithms can provide an efficient framework for the design of 5G Intelligent Networks.


Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !
Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email, explore the Big Picture in Compressive Sensing or the Matrix Factorization Jungle and join the conversations on compressive sensing, advanced matrix factorization and calibration issues on Linkedin.

Tuesday, January 09, 2018

Video Streaming: State of Artificial Intelligence for Environmental Science Applications panel at AMS conference in Austin TX



Philippe tells me that, at the American Meteorological Society annual meeting in Austin, TX, there will be a broadcast of the State of Artificial Intelligence for Environmental Science Applications panel, today (Tuesday) at 2pm -3pm (CST) - or 12pm-1pm PST, 3pm-4pm EST, 9pm - 10pm Paris time. The panel will be broadcast on YouTube. If you are attending remotely you can tweet questions to  @AMS_AIML.



More about the Panel:

“A panel of international experts on Artificial Intelligence (AI) techniques applied to Environmental Sciences, including government, academic and private sector, will discuss the role of AI on current applications, and will provide their perspectives on the challenges and potential applications of novel AI methods. The panel will discuss regression and classification-based approaches, as well as examples of supervised and unsupervised learning problems. Specific examples, best practices and lessons learned will be highlighted. Outcomes from the session will be used to provide a framework to facilitate the integration of Artificial Intelligence and Environmental Science research initiatives.”






Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !

Thursday, January 04, 2018

Ce soir: Paris Machine Learning meetup Hors Série 1 (Season 5) Self-Supervised Imitation, Pierre Sermanet




Video streaming starts at 6:30PM Paris time.




Pierre Sermanet is our guest for the first Paris Machine Learning Hors Série of the season. The talk should be about 30-45 minutes long. Thanks to Meritis for hosting this meetup and providing the food and drinks afterwards. Doors open 6:15PM, talk will go from 6:30-7:15PM and will have food and drinks from 7:15 till 8:15PM Paris time.

Title: Self-Supervised Imitation (presentation slide are 400MB large)
Abstract: We propose a self-supervised approach for learning representations and robotic behaviors entirely from unlabeled videos recorded from multiple viewpoints. We study how these representations can be used in two robotic imitation settings: imitating object interactions from videos of humans, and imitating human poses. Imitation of human behavior requires a viewpoint-invariant representation that captures the relationships between end-effectors (hands or robot grippers) and the environment, object attributes, and body pose. We train our representations using a triplet loss, where multiple simultaneous viewpoints of the same observation are attracted in the embedding space, while being repelled from temporal neighbors which are often visually similar but functionally different. This signal causes our model to discover attributes that do not change across viewpoint, but do change across time, while ignoring nuisance variables such as occlusions, motion blur, lighting and background. We demonstrate that this representation can be used by a robot to directly mimic human poses without an explicit correspondence, and that it can be used as a reward function within a reinforcement learning algorithm. While representations are learned from an unlabeled collection of task-related videos, robot behaviors such as pouring are learned by watching a single 3rd-person demonstration by a human. Reward functions obtained by following the human demonstrations under the learned representation enable efficient reinforcement learning that is practical for real-world robotic systems. Video results, open-source code and dataset are available at https://sermanet.github.io/imitate






Join the CompressiveSensing subreddit or the Google+ Community or the Facebook page and post there !

Printfriendly