RING: Repair Is Nearly Generation: Multilingual Program Repair with LLMs (Microsoft)

An LLMC-based approach to multilingual repair powered by Codex. It enables a flipped model for AI-assisted programming, in which the user writes code and the assistant suggests fixes for last-mile mistakes. The approach is built on the intuition that fixing such mistakes can be conceptualized into three tasks: fault localization, code transformation, and candidate ranking.

Transformers in Time Series: A Survey

The application of the Transformer architecture looks natural in the case of time series. In a sense, time series is even a more native area for sec2sec approaches than CV or NLP. What peculiarities of time series should be taken into account when using Transformers? A lot of them. Let's list some.
- it is often necessary to be able to process very long sequences (periodicity, seasonality)
- sequences in time series often have timestamps labels that contain more information than just a position number in the sequence from NLP or CV. For example, days of the week, holidays. - sometimes sequential generation is not efficient and the decoder needs to be able to generate subsequences at one time
- time series pose special tasks: spatio-temporal forecasting, event forecasting, anomaly detection, etc.

The survey systematizes approaches in the field of application of Transformers for time series. In addition, the authors test different algorithms on the ETTm2 dataset.

Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting

As you know, there are several severe issues with Transformer that prevent it from being directly applicable to long sequence time-series forecasting including:
- quadratic time complexity
- high memory usage
- inherent limitation of the encoder-decoder architecture.
To address these issues, the authors design an efficient transformer-based model named Informer.

Informer includes:
- a ProbSparse self-attention mechanism, which achieves O(L log L) in time complexity and memory usage, and has comparable performance on sequences’ dependency
alignment
- the self-attention distilling highlights dominating attention by halving cascading layer input, and efficiently handles extreme long input sequences
- the generative style decoder, while conceptually simple, predicts the long time-series sequences at one forward operation rather than a step-by-step way, which drastically improves the inference
speed of long-sequence predictions.

COMBO Pre-training representations of binary code using contrastive learning

Binary code analysis is critical to applications in reverse engineering and computer security tasks where source code is not available.

In the paper authors propose a COntrastive learning Model for Binary cOde Analysis, or COMBO, that incorporates source code and comment information into binary code during representation learning. Specifically, there are three components in COMBO:
- a primary contrastive learning method for cold-start pre-training
- a simplex interpolation method to incorporate source code, comments, and binary code
- an intermediate representation learning algorithm to provide binary code embeddings.

The effectiveness of the pre-trained representations produced by COMBO is evaluated using three indicative downstream tasks relating to binary code:
- algorithmic functionality classification
- binary code similarity
- vulnerability detection.

UL2: Unifying Language Learning Paradigms (Google)

A novel language pre-training paradigm called Unified Language Learner (UL2) frames different objective functions for training language models as denoising tasks, where the model has to recover missing sub-sequences of a given input. During pre-training it uses a novel mixture-of-denoisers that samples from a varied set of such objectives, each with different configurations.

Forest: Structural Code Editing with Multiple Cursors (ETH)

Forest allows to perform a single action simultaneously in multiple program locations and thus support complex refactorings.

Minerva: Solving Quantitative Reasoning Problems with Language Models (Google)

Minerva is a large language model pretrained on general natural language data and further trained on technical content. The main novelty of the paper is a large training dataset that juxtaposes natural language with the correct use of formal mathematical language, such as equations and diagrams. The data is collected from the arXiv preprint server and from web pages.

Code as Policies: Language Model Programs for Embodied Control (Google)

Large language models trained on code completion have been shown to be capable of synthesizing simple Python programs from docstrings. These models can be re-purposed to write robot policy code, given natural language commands.

Project website: https://code-as-policies.github.io/

Research talk: Cloud Intelligence/AIOps – Infusing AI into cloud computing (Microsoft)

Fixing Dockerfile Smells: An Empirical Study

RQ1: How do developers fix Dockerfile smells?
RQ2: Which Dockerfile smells are developers willing to address?