Categories Machine Learning

How to simply generate Smart Replies with ML Kit on Android

7 Apr. 2019
3
0
15 minutes
ML Kit Smart Replies

Nowadays, Machine Learning is all around us. From predicting which movies you may like, to tagging persons in a picture, to predict fraud while paying. This branch of Artificial Intelligence lets computer algorithms learn automatically through experience. Machine Learning is a breakthrough in the technology world and it is becoming more and more widespread. The mobile industry is also affected by this phenomenon. It allows mobile developers to enhance drastically the user experience on their apps.

Natural Language Processing is a field in Machine Learning that deals with analysis, manipulation, understanding and generation of Natural Language, that is, human language. There are various applications for Natural Language Processing: sentiment analysis, classification, speech recognition, question answering, etc. Some popular use cases are assistants (Google Assistant, Siri), chatbots that answer your questions and translators such as Google Translate.

Google provides a powerful mobile SDK, ML Kit, which allows Android developers to implement Machine Learning functionalities easily. This SDK gathers several Google’s ML technologies, such as the Google Cloud Vision APITensorFlow Lite, and the Android Neural Networks API . ML Kit comes with a set of ready-to-use APIs for common mobile use cases such as:

  • Text recognition
  • Barcode scanning
  • Image labelling
  • Language identification
  • Face detection
  • Smart replies
  • Landmark identification

Introduction

This post focuses on Smart Reply API. This is a powerful API that can automatically generate relevant reply suggestions to messages based on the full context of a conversation and not a single message. Smart Replies help users respond to messages more quickly. It also helps users whose devices have limited input capabilities to reply to messages in an easier manner. Some apps already use a model to generate reply suggestions. For instance, Gmail, Google Assistant and Google Allo use such a model.

The screenshot below demonstrates how Smart Reply API can be used.

Smart Reply demonstration

Demonstration

Model operation

The Machine Learning model used by the Smart Reply API detects the language of the conversation and tries to generate reply suggestions only if the language is determined to be English. The model uses up to 10 of the last message from a conversation history to generate reply suggestions. The next step aims at analyzing conversation and detecting sensitive topics. Reply suggestions aren’t generated when the conversation includes a sensitive topic to avoid inappropriate suggestions in case of personal hardship or profanity. If the conversation is in English and doesn’t contain sensitive topics, the model provides up to three reply suggestions. The number of suggestions depends on the level of confidence of the model for reply suggestion generation in a given context.

The following screenshot shows that no reply suggestion is generated in the case of a conversation containing a sensitive topic.

Conversation with a sensitive topic

Limitations

For now, Smart Reply API has limited capabilities. First, only English is supported yet. The Machine Learning model used by the API automatically detects if a non-English language is used and if so, will not provide reply suggestions. Furthermore, it is intended for casual conversations in consumer apps. Reply suggestions might not be appropriate for other contexts or audiences.

Implementation

This post gets right to the point and describes how ML Kit Smart Reply works and how it can be implemented. However, a complete project using this feature is on Github, the links are at the bottom of the page. This project simulates a conversation between a local user and a remote user. You can switch between users at any time (first app bar button) to answer for the remote user in order to test ML Kit Smart Reply in depth. You can clear the conversation using the second button of the app bar. Here is another screenshot of the project:

Smart Reply Project demonstration

To integrate ML Kit Smart Reply in your application, you must first add Firebase and ML Kit Android libraries to your project. Then a conversation history must be created and passed to ML Kit in order to generate reply suggestions.

Firebase integration

As ML Kit is part of Firebase, you must add Firebase to your project to use Smart Reply API. In order to achieve this, a step-by-step guide is available on Firebase website.

Setup

Once your project includes Firebase, ML Kit Android libraries must be included in your project. To do so, include dependencies in your app-level build.gradle file as follows:

dependencies {
  // ...

  implementation 'com.google.firebase:firebase-ml-natural-language:18.2.0'
  implementation 'com.google.firebase:firebase-ml-natural-language-smart-reply-model:18.0.0'
}

Then disable compression of tflite files in your app-level build.gradle file.
This prevents Gradle compressing the models when building the app.

android {
    // ...
    aaptOptions {
        noCompress "tflite"
    }
}

Conversation history

There is no possible reply suggestion without a conversation history. Therefore, the first step consists in creating a conversation history which will serve as a context for ML Kit Smart Reply API.

ML Kit expects a list of FirebaseTextMessage objects that is chronologically-ordered with the earliest timestamp first. You can create such objects using one of the two static methods:

Therefore, whenever the local user sends a message, add its content and its timestamp to the conversation history as follows:

val chatHistory = ArrayList<FirebaseTextMessage>()

chatHistory.add(
    FirebaseTextMessage.createForLocalUser(
        "Do you want to watch a movie tonight ?",
        System.currentTimeMillis()
    )
)
List<FirebaseTextMessage> chatHistory = new ArrayList<>();

chatHistory.add(FirebaseTextMessage.createForLocalUser(
        "Do you want to watch a movie tonight ?",
        System.currentTimeMillis()));

Whenever the local user receives a message from a remote user, you should also add its content and its timestamp as well as the sender’s user id to the conversation history. Indeed this will help ML Kit to understand the context and provide relevant reply suggestions.

chatHistory.add(FirebaseTextMessage.createForRemoteUser(
        "Do you want to watch a movie tonight ?",
        System.currentTimeMillis(),
        remoteUserId));
chatHistory.add(FirebaseTextMessage.createForRemoteUser(
        "Do you want to watch a movie tonight ?",
        System.currentTimeMillis(),
        remoteUserId));

The user ID is a string that uniquely identifies the sender within the conversation.

Generation of reply suggestions

Once the conversation history is created, you can attempt to get reply suggestions to the latest received message by passing the chat history to ML Kit. In order to achieve this, the first step consists in obtaining an instance of FirebaseSmartReply. Then you have to pass the conversation history to ML Kit using suggestReplies(List<FirebaseTextMessage> textMessages) method.

If the operation succeeds, a SmartReplySuggestionResult is passed to the success handler. You can determine if the language of the conversation is supported and if it isn’t, the suggestion list will be empty. If the language is supported, you can get the results by calling getSuggestions() method on the result object. Then you can loop through the reply suggestion list and do whatever you want with it.

Note that ML Kit might not return results if the model isn’t confident in the relevance of the suggested replies.
FirebaseNaturalLanguage.getInstance().smartReply.suggestReplies(chatHistory)
    .addOnSuccessListener { result ->
        if (result.status == SmartReplySuggestionResult.STATUS_NOT_SUPPORTED_LANGUAGE) {
            // The conversation's language isn't supported, so the
            // the result doesn't contain any suggestions.
        } else if (result.status == SmartReplySuggestionResult.STATUS_SUCCESS) {
            // Task completed successfully
            val suggestions = result.suggestions
            // Use reply suggestions as you wish
        }
    }
    .addOnFailureListener {
        // Task failed with an exception
    }
FirebaseNaturalLanguage.getInstance().getSmartReply().suggestReplies(chatHistory)
        .addOnSuccessListener(new OnSuccessListener<SmartReplySuggestionResult>() {
            @Override
            public void onSuccess(SmartReplySuggestionResult result) {
                if (result.getStatus() == SmartReplySuggestionResult.STATUS_NOT_SUPPORTED_LANGUAGE) {
                    // The conversation's language isn't supported, so the
                    // the result doesn't contain any suggestions.
                } else if (result.getStatus() == SmartReplySuggestionResult.STATUS_SUCCESS) {
                    // Task completed successfully
                    List<SmartReplySuggestion> suggestions = result.getSuggestions();
                    // Use reply suggestions as you wish
                }
            }
        })
        .addOnFailureListener(new OnFailureListener() {
            @Override
            public void onFailure(@NonNull Exception e) {
                // Task failed with an exception
                // ...
            }
        });

Download

  • Kotlin
  • Java

Conclusion

In conclusion, ML Kit is a powerful tool that increases drastically opportunities for developers. ML Kit Smart Reply is a useful API that lets developers generate reply suggestions in a conversation. This feature helps users to answer more quickly to their contact by using relevant reply suggestions based on the conversation context.

I think that ML Kit Smart Reply will be used in a lot of apps, such as messaging apps, mailing apps, assistant apps, etc. And you? How would you use the power of Smart Reply? Feel free to share your thoughts.