Those pits of Gson serialization and deserialization after confusion in android development

Those pits of Gson serialization and deserialization after confusion in android development

Serialization and deserialization of GSON

GSON is a very good tool, using it we can easily achieve serialization and deserialization. But once it encounters confusion, we need to pay attention.

A simple class Item, used to handle serialization and deserialization

public class Item { public String name; public int id; } Copy code

Serialized code

Item toSerializeItem = new Item(); toSerializeItem.id = 2; toSerializeItem.name = "Apple"; String serializedText = gson.toJson(toSerializeItem); Log.i(LOGTAG, "testGson serializedText=" + serializedText); Copy code

Log output after obfuscation is turned on

I/MainActivity: testGson serializedText = { "a": "Apple", "b": 2} copy the code

The attribute name has been changed and it has become a meaningless name, which is very troublesome for some of our subsequent processing.

Deserialization code

Gson gson = new Gson(); Item item = gson.fromJson("{\"id\":1,/"name\":\"Orange\"}", Item.class); Log.i(LOGTAG, "testGson item.id=" + item.id + ";item.name=" + item.name); Copy code

The corresponding log result is

I/MainActivity: testGson item.id = 0 ; item.name = null duplicated code

It can be seen that after the confusion, the deserialized attribute value settings have failed.

why?

  • Because the nature of the object created by deserialization still uses reflection, the key of the json string will be used as the attribute name, and the value will correspond to the attribute value.

How to solve

  • Eliminate confusion between serialized and deserialized classes
  • Annotate fields with @SerializedName

@SerializedName(parameter) is implemented through annotated attributes

  • In the serialized result, specify the attribute key as the value of parameter.
  • In the object generated by deserialization, it is used to match key and parameter and assign attribute values.

A simple usage is

public class Item { @SerializedName("name") public String name; @SerializedName("id") public int id; } Copy code

Of course, you can also write directly like this

public class Item implements Serializable{ public String name; public int id; } Copy code

The above is the problem of deserialization after the confusion is realized by adding the @SerializedName annotation. Next, we will talk about how to eliminate the confusion of the serialization and deserialization classes.

package com.baidu.bean; public class Item { public String name; public int id; public static class PageConfig { public String type; } } Copy code

An internal class PageConfig has been added to Item.

Knock on the blackboard here :

1. The fields in the Item, the classes referenced in the Item, and the internal class PageConfig in the Item need to implement serializable (implements Serializable);

  1. If instead of implementing Serializable to implement serialization, but adding @SerializedName annotations to each field, be sure to note: @SerializedName must be added to the fields in the Item, the classes referenced in the Item, and the fields of the internal classes in the Item Note, otherwise there will be inexplicable problems: it will not crash, it is all kinds of strange phenomena, and this problem does not appear under debug.

The most common practice is:

-keep class com.baidu.bean.** {*; } Copy code

The meaning is: exclude classes under the bean directory including subdirectories not to be confused

To exclude a class individually, you can write:

-keep class com.baidu.bean {*;}

To exclude the inner class of a certain class separately, you need to write:

-keep class class com.baidu.bean.Item$PageConfig {*;}

If there are internal classes in many entity classes, it is recommended to write in combination:

-keep class com.baidu.bean.**{ *;} -keep class com.baidu.bean.**$*{ *;} Copy code

In addition, the following writing is also possible, mainly based on the above writing. Readers can use which one to use according to their needs.

-keep class com.baidu.bean. $** { ;}

-keep class com.baidu.bean. $ {*;}

-keep class com.baidu.bean.**$* {*;}

The configuration of * and ** wildcards appears above, in order to deepen the impression, here is an extended reading:

Android obfuscation best practices

1. Confusion configuration

android{ buildTypes { release { buildConfigField "boolean", "LOG_DEBUG", "false"//Do not display log minifyEnabled true shrinkResources true proguardFiles getDefaultProguardFile('proguard-android.txt'),'proguard-rules.pro' signingConfig signingConfigs.config } } } Copy code

Because turning on obfuscation will make the compilation time longer, it is not turned on in debug mode. What we need to do is:

  1. will
    release minifyEnabled
    Changed to
    true
    , Turn on confusion;
  2. Plus
    shrinkResources true
    To turn on resource compression.

3.

buildConfigField
 No log log
4.
signingConfig signingConfigs.config
Configure signature file

Custom obfuscation rules

Custom obfuscation scheme is suitable for most projects

#Specify the compression level -optimizationpasses 5 #Do not skip non-public library class members -dontskipnonpubliclibraryclassmembers #Algorithm used when obfuscating -optimizations !code/simplification/arithmetic,!field/*,!class/merging/* #The method name in the obfuscated class is also confused -useuniqueclassmembernames #Optimization allows access and modification of classes and class members with modifiers -allowaccessmodification #Rename the source of the file to the string "SourceFile" -renamesourcefileattribute SourceFile #Keep line number -keepattributes SourceFile,LineNumberTable #Keep generic -keepattributes Signature #Keep all class members that implement the Serializable interface -keepclassmembers class * implements java.io.Serializable { static final long serialVersionUID; private static final java.io.ObjectStreamField[] serialPersistentFields; private void writeObject(java.io.ObjectOutputStream); private void readObject(java.io.ObjectInputStream); java.lang.Object writeReplace(); java.lang.Object readResolve(); } #Fragment does not need to be registered in AndroidManifest.xml, it needs additional protection -keep public class * extends android.support.v4.app.Fragment -keep public class * extends android.app.Fragment # Keep test-related code -dontnote junit.framework.** -dontnote junit.runner.** -dontwarn android.test.** -dontwarn android.support.test.** -dontwarn org.junit.** Copy code

The really common ones that need to be added are the above. In addition, each project needs to add some obfuscation rules according to its own needs:

Obfuscation rules required by third-party libraries. Regular third-party libraries generally write the required obfuscation rules in the access documents, and pay attention to adding them when using them.

Code that changes dynamically at runtime, such as reflection. A typical example is the entity class that will convert to and from json. If the project naming convention requires entity classes to be placed under the model package, you can add code like this to keep all entity classes:

-keep public class **.*Model*.** {*;}

The class called in JNI.

WebView

JavaScript
Method called

Layout The View constructor used by the layout,

android:onClick
Wait.

Check the result of confusion

Obfuscated packages must be checked to avoid bugs introduced by obfuscation.

On the one hand, it needs to be checked from the code level. Use the above configuration to obfuscate and package in

<module-name>/build/outputs/mapping/release/
The following files will be output in the directory:

dump.txt

Describe the internal structure of all classes in the APK file

mapping.txt

Provide a comparison table of classes, methods, class members, etc. before and after confusion

seeds.txt

List classes and members that are not confused

usage.txt

List removed codes

We can follow 

seeds.txt
 The file checks whether the classes and members that are not obfuscated contain all the expected reservations, and then according to 
usage.txt
Check the file for code that was removed by mistake.

On the other hand, it needs to be checked from the test. The obfuscated package will be tested in all aspects to check whether there is a bug.

Solve the confusion stack

Obfuscated classes, method names, etc. are difficult to read, which will increase the difficulty of reverse engineering, but it also hinders the tracking of online crashes. After we get the stack information of the crash, we will find it difficult to locate it. At this time, we need to reverse the confusion.

in 

<sdk-root>/tools/proguard/
There is an accompanying reverse solution tool under the path (Window system is
proguardgui.bat
, Mac or Linux system is
proguardgui.sh
).

Take the Window platform as an example here. After double-clicking to run proguardgui.bat, you can see a row of menus on the left. Click ReTrace and select the mapping file corresponding to the obfuscated package (after obfuscation, the mapping.txt file will be generated under the/build/outputs/mapping/release/path. Its function is to provide a comparison table of classes, methods, class members, etc. before and after obfuscation ), then paste the crash stack trace into the input box, click ReTrace in the lower right corner, and the confused stack information will be displayed.

The above uses the GUI program to operate, another way is to use the retrace tool under the path to reverse the solution through the command line. The command is

retrace.bat|retrace.sh [-verbose] mapping.txt [<stacktrace_file>]

E.g:

retrace.bat -verbose mapping.txt obfuscated_trace.txt

Precautions:

All in 

AndroidManifest.xml
 The classes involved have been automatically maintained, so there is no need to add this obfuscation rule. (Many old obfuscated files will be added, it is no longer necessary)

proguard-android.txt
There are already some default obfuscation rules, there is no need to add them repeatedly in proguard-rules.pro

Introduction to Obfuscation

The "obfuscation" in Android can be divided into two parts, one part is

Java
 Optimization and confusion of the code, rely on 
proguard
The obfuscator is implemented; the other part is resource compression, which will remove unused resources in the project and dependent libraries (resource compression has nothing to do with obfuscation in the strict sense, but generally we will use them together).

Code compression

Code compression process

Code obfuscation is a process involving a series of behaviors such as code compression, optimization, and obfuscation. As shown in the figure above, the obfuscation process will have the following functions:

compression. Remove invalid classes, class members, methods, attributes, etc.;
optimize. Binary code of analysis and optimization method; according to

proguard-android-optimize.txt
As described in, optimization may cause some potential risks, and it cannot guarantee normal operation on all versions of Dalvik.
Confused. Replace class names, attribute names, and method names with short and meaningless names;
pre-verification. Add pre-check information. This pre-check works on the Java platform. This function is not needed on the Android platform. After removing it, the obfuscation speed can be accelerated.
These four processes are turned on by default.

in

Android
In the project, we can choose to turn off "optimization" and "pre-check", the corresponding command is
-dontoptimize, -dontpreverify
(Of course, the default 
proguard-android.txt
The file already contains these two obfuscated commands, no additional configuration by the developer is required).

Resource compression

Resource compression will remove unused resources in the project and dependent libraries, which is reducing

apk
 The package size will have a good effect, and it is generally recommended to turn it on. The specific approach is 
build.grade
In the file, add
shrinkResources
Attribute is set to
true
. It should be noted that only in use
minifyEnabled true
After the code compression is turned on, the resource compression will take effect.

Resource compression includes the two processes of "merging resources" and "removing resources".

In the process of "merging resources", resources with the same name are treated as duplicate resources and will be merged. It s important to note that this process is not

shrinkResources
Attribute control can t be banned,
gradle
This work must be done, because if resources with the same name exist in different projects, it will cause an error.
gradle
 Look for duplicate resources everywhere:

src/main/res/
 path
different build types (
debug
,
release
Etc.)
Different build channels
The third-party libraries that the project depends on.
When merging resources, follow the priority order as follows

Dependent -> main -> Channel -> Construction type copy the code

If duplicate resources exist in the main folder and different channels at the same time, gradle will choose to keep the resources in the channel.

At the same time, if duplicate resources appear at the same level, such as

src/main/res/and src/main/res2/
,then 
gradle
The resource merging cannot be completed, and a resource merging error will be reported.

The process of "remove resources" is as the name implies. It should be noted that, similar to the code, obfuscated resource removal can also define which resources need to be reserved. This is given below.

Custom obfuscation rules

There is such a line of code in the "obfuscated configuration" above

proguardFiles getDefaultProguardFile('proguard-android.txt'),'proguard-rules.pro' Copy code

This line of code defines that the obfuscation rule consists of two parts: the content of proguard-android.txt located in the tools/proguard/folder of the SDK and the content of proguard-rules.pro that is placed in the root directory of the module by default. The former is the default obfuscation file provided by the SDK, and the latter is where the developer customizes the obfuscation rules.

Common obfuscated instructions

  • optimizationpasses
  • dontoptimize
  • dontusemixedcaseclassnames
  • dontskipnonpubliclibraryclasses
  • dontpreverify
  • dontwarn
  • verbose
  • optimizations
  • keep
  • keepnames
  • keepclassmembers
  • keepclassmembernames
  • keepclasseswithmembers
  • keepclasseswithmembernames for
    more details, please go to the official website

What needs special introduction are several commands related to the rules to keep related elements from participating in obfuscation:

commandeffect
-keepPrevent classes and members from being removed or renamed
-keepnamesPrevent classes and members from being renamed
-keepclassmembersPrevent members from being removed or renamed
-keepnamesPrevent members from being renamed
-keepclasseswithmembersPrevent classes and members that have the member from being removed or renamed
-keepclasseswithmembernamesPrevent classes and members that have the member from being renamed

Rules to keep elements from participating in obfuscation

[Hold order] [class] { [member] } Copy code

"Class" represents class-related qualifications, and it will eventually locate certain classes that meet the qualifications. Its content can be used:

  • Concrete class
  • Access modifiers (public, protected, private)
  • Wildcard *, matches characters of any length, but does not contain the package name separator (.)
  • Wildcard **, matches characters of any length, and includes the package name separator (.)
  • extends, which can specify the base class of the class
  • implement, to match the class that implements an interface
  • $, inner class

"Member" represents the qualifications related to class members, and it will eventually locate some class members that meet the qualifications. Its content can be used:

  • Match all constructors
  • Match all domains
  • Match all methods
  • Wildcard *, matches characters of any length, but does not contain the package name separator (.)
  • Wildcard **, matches characters of any length, and includes the package name separator (.)
  • Wildcard ***, matches any parameter type
  • , match any type parameter of any length. For example, void test(...) can match any void test(String a) or void test(int a, String b) methods.
  • Access modifiers (public, protected, private)
    For example, if you need to keep all public classes and their constructors that inherit Activity under the com.biaobiao.test package, you can write:
-keep public class com.biaobiao.test.** extends Android.app.Activity { <init> } Copy code

Commonly used custom obfuscation rules

  • Do not confuse a certain class
-keep public class com.biaobiao.example.Test {*;} Copy code

Do not confuse all classes in a package

-keep class com.biaobiao.test.** {*;} } Copy code

Do not confuse the subclasses of a certain class

-keep public class * extends com.biaobiao.example.Test {*;} Copy code

Do not confuse all classes and their members that contain "model" in the class name

-keep public class * extends com.biaobiao.example.Test {* ;} copy the code

Do not confuse the implementation of an interface

-keep class * implements com.biaobiao.example.TestInterface {*; } copy the code

Do not confuse the construction method of a class

-keepclassmembers class com.biaobiao.example.Test { public <init>(); } Copy code

Do not confuse specific methods of a certain class

-keepclassmembers class com.biaobiao.example.Test { public void test(java.lang.String); } } Copy code

Do not confuse the inner classes of a certain class

-keep class com.biaobiao.example.Test$* { *; } Copy code

Custom resource retention rules

1. keep.xml

use

shrinkResources true
After enabling resource compression, all unused resources are removed by default. If you need to define which resources must be reserved, in 
res/raw/
 Create an xml file under the path, for example
keep.xml
 .

The requirements of defining resource retention can be achieved through the setting of some attributes. The configurable attributes are:

  • keep
     Define which resources need to be reserved (separate resources with ",")
  • discard
     Define which resources need to be removed (separate resources with ",")
  • shrinkMode
     Turn on strict mode
  • When the code passes
    Resources.getIdentifier()
     When using dynamic strings to obtain and use resources, ordinary resource reference checks may be problematic. For example, the following code will cause all resources beginning with "img_" to be marked as used.
    When the code passes 
    Resources.getIdentifier()
     When using dynamic strings to obtain and use resources, ordinary resource reference checks may be problematic. For example, the following code will cause all resources beginning with "img_" to be marked as used.
String name = String.format("img_%1d", angle + 1); res = getResources().getIdentifier(name, "drawable", getPackageName()); Copy code

We can set 

tools:shrinkMode
 for
strict
To enable strict mode, only the resources that are actually used are reserved.

The above is the configuration related to the custom resource retention rule, for example:

<?xml version="1.0" encoding="utf-8"?> <resources xmlns:tools="http://schemas.android.com/tools" tools:keep="@layout/l_used*_c,@layout/l_used_a,@layout/l_used_b*" tools:discard="@layout/unused2" tools:shrinkMode="strict"/> Copy code

Remove alternative resources

Some alternative resources, such as strings.xml for multi-language support, layout.xml for multi-resolution support, etc., can be removed using resource compression when we do not need to use them and do not want to delete them.

We use the resConfig attribute to specify the attributes that need to be supported, such as
some alternative resources, such as multi-language support

strings.xml
, Multi-resolution support 
layout.xml
Wait, when we don't need to use and don't want to delete them, we can use resource compression to remove them.

We use 

resConfig
Attributes to specify the attributes that need to be supported, for example

android { defaultConfig { ... resConfigs "en", "fr" } } Copy code

Other language resources that are not explicitly declared will be removed.

Finally, I attach an obfuscation scheme in the actual project

proguard-android.txt
document content

# Code obfuscation compression ratio, between 0~7 -optimizationpasses 5 # When mixing, do not use mixed case, the mixed class name is lowercase -dontusemixedcaseclassnames # Specify not to ignore non-public library classes -dontskipnonpubliclibraryclasses # No pre-verification, preverify is one of the four steps of proguard, Android does not need preverify, removing this step can speed up the confusion. -dontpreverify -verbose # Avoid confusion about generics -keepattributes Signature # Keep Annotation not to be confused -keepattributes *Annotation*,InnerClasses #googleRecommendation Algorithm -optimizations !code/simplification/arithmetic,!code/simplification/cast,!field/*,!class/merging/* # Avoid confusion of Annotation, inner classes, generics, anonymous classes -keepattributes *Annotation*,InnerClasses,Signature,EnclosingMethod # Rename the file name when the exception was thrown -renamesourcefileattribute SourceFile # Keep the code line number when throwing an exception -keepattributes SourceFile,LineNumberTable # Processing support package -dontnote android.support.** -dontwarn android.support.** # Keep inherited -keep public class * extends android.support.v4.** -keep public class * extends android.support.v7.** -keep public class * extends android.support.annotation.** # Reserve the resources under R -keep class **.R$* {*;} # Keep the four major components, custom Application and other classes will not be confused -keep public class * extends android.app.Activity -keep public class * extends android.app.Appliction -keep public class * extends android.app.Service -keep public class * extends android.content.BroadcastReceiver -keep public class * extends android.content.ContentProvider -keep public class * extends android.preference.Preference -keep public class com.android.vending.licensing.ILicensingService # The method parameter retained in the Activity is the view method, # So the onClick we wrote in the layout will not be affected -keepclassmembers class * extends android.app.Activity{ public void *(android.view.View); } # For onXXEvent and **On*Listener with callback functions, they cannot be confused -keepclassmembers class * { void *(**On*Event); void *(**On*Listener); } # Keep local native methods from being confused -keepclasseswithmembernames class * { native <methods>; } # Keep the enumeration class from being confused -keepclassmembers enum * { public static **[] values(); public static ** valueOf(java.lang.String); } # Keep the Parcelable serialization class from being confused -keep class * implements android.os.Parcelable { public static final android.os.Parcelable$Creator *; } -keepclassmembers class * implements java.io.Serializable { static final long serialVersionUID; private static final java.io.ObjectStreamField[] serialPersistentFields; private void writeObject(java.io.ObjectOutputStream); private void readObject(java.io.ObjectInputStream); java.lang.Object writeReplace(); java.lang.Object readResolve(); } #assume no side effects: delete the log output by android.util.Log -assumenosideeffects class android.util.Log { public static *** v(...); public static *** d(...); public static *** i(...); public static *** w(...); public static *** e(...); } #Keep annotated class names and methods -keep,allowobfuscation @interface android.support.annotation.Keep -keep @android.support.annotation.Keep class * -keepclassmembers class * { @android.support.annotation.Keep *; } #3D Map before V5.0.0: -dontwarn com.amap.api.** -dontwarn com.autonavi.** -keep class com.amap.api.**{*;} -keep class com.autonavi.**{*;} -keep class com.amap.api.maps.**{*;} -keep class com.autonavi.amap.mapcore.*{*;} -keep class com.amap.api.trace.**{*;} #3D Map after V5.0.0: -keep class com.amap.api.maps.**{*;} -keep class com.autonavi.**{*;} -keep class com.amap.api.trace.**{*;} #Positioning -keep class com.amap.api.location.**{*;} -keep class com.amap.api.fence.**{*;} -keep class com.autonavi.aps.amapapi.model.**{*;} #search for -keep class com.amap.api.services.**{*;} #2DMap -keep class com.amap.api.maps2d.**{*;} -keep class com.amap.api.mapcore2d.**{*;} #navigation -keep class com.amap.api.navi.**{*;} -keep class com.autonavi.**{*;} # Retain generic type information for use by reflection by converters and adapters. -keepattributes Signature # Retain service method parameters when optimizing. -keepclassmembers,allowshrinking,allowobfuscation interface * { @retrofit2.http.* <methods>; } # Ignore annotation used for build tooling. -dontwarn org.codehaus.mojo.animal_sniffer.IgnoreJRERequirement # Ignore JSR 305 annotations for embedding nullability information. -dontwarn javax.annotation.** # JSR 305 annotations are for embedding nullability information. -dontwarn javax.annotation.** # A resource is loaded with a relative path so the package of this class must be preserved. -keepnames class okhttp3.internal.publicsuffix.PublicSuffixDatabase # Animal Sniffer compileOnly dependency to ensure APIs are compatible with older versions of Java. -dontwarn org.codehaus.mojo.animal_sniffer.* # OkHttp platform used only on JVM and when Conscrypt dependency is available. -dontwarn okhttp3.internal.platform.ConscryptPlatform #fastjson obfuscation -keepattributes Signature -dontwarn com.alibaba.fastjson.** -keep class com.alibaba.**{*;} -keep class com.alibaba.fastjson.**{*;} -keep public class com.ninstarscf.ld.model.entity.**{*;} Copy code