Walking the Android (time)line Part 2 – Using Android’s Device Personalization Services to timeline user activity

Dying
I almost died.

There was a purpose behind the above Tweet. When I was looking in my Android 10 image I actually found two features that were tracking my activity (there may be more – I haven’t finished looking yet). The first, Digital Wellbeing, had a UI and was fairly straight forward to test, so I decided to work on that one first. The second feature had no on-device UI and looked a bit more complicated. I thought it would require a little more work just to test, so I decided to make Android my daily driver for a month….all in the name of research. While not completely excruciating, I was on the struggle bus for about a week trying to get used to the UI and finding the Android equivalents of iOS things. Nothing worth having is ever easy, right?

Back in February I wrote a blog post about using Digital Wellbeing to track user activity on an Android device. This blog post will address the second feature: Device Personalization Services (DPS). DPS is different than Digital Wellbeing in a few notable ways, the most important of which is that it retains data about activity from deleted apps.

Tracking user activity on a mobile device can be helpful in so many ways, and can be powerful when documented in a timeline. It can prove someone was/was not using their device at a certain time (or in/not in a certain place). It can go towards corroborating someone’s account of events, or to disprove those accounts. It can add context to a particular datapoint. The use cases are almost limitless. So, being able to use DPS (and other features like it) to get a better understanding of user activity within context, including in apps that have been deleted, can be a forensic goldmine.

Background

DPS started out as Action Services, and was introduced as App Actions at Google I/O 2018 alongside the Android 9 beta, although Google was quick to point out it was not limited to that OS version. The service was (and is) integrated into Android and the wider Google ecosystem, and is designed to learn how a person uses their device and apps (i.e. their habits) and, based on that information, recommend actions to a user based on context (location, time of day, app usage and sequence). These suggestions could appear in the App Drawer, on the device’s lock screen, or within a contextual menu inside of an app. I suspect some of the data is collected via the Firebase app indexing API (specifically, the usage logging portion), but that is strictly speculation.

When Android 10 arrived Action Services was re-branded as DPS, but it kept the same folder name in the Android file system. In addition to the new name, DPS received a privacy-focused feature: the ability to delete the data it collected. This feature is in the new Privacy menu at the top of the Settings app.  See Figure 1.

Figure 1
Figure 1.  Controlling the (your) data.

Users have the option to delete DPS data from the last hour, the last 24 hours, or all of the data.

It would not be surprising if one felt like DPS is somewhat an enigma; it is to some degree. The closest thing that I could find to a definition came from 9to5Google:

Device Personalization Services uses system permissions to provide smart predictions. For example, you may see a suggestion to call a frequent contact or return a missed call. It uses your location to link you to the right nearby restaurant when you select its name. Device Personalization Services is part of the [sic] and you can control it in Home settings.

At I/O 2019 Sundar Pichai said Google would be pushing machine learning out to the edge and it looks as if this is holding true; on-device machine learning is a feature for Android 10 (Live Captions and Smart Reply) and I believe DPS is part of the initiative. If this on-device machine learning shtick sounds familiar it should because there is an iOS equivalent. Siri Suggestions does the same thing. Siri uses the myriad of databases and plists in iOS and some fancy (supposedly) on-device machine learning to make “intelligent” suggestions to users, and I think DPS is trying…trying…to do the same thing here.

The Basics

DPS resides in the /data/data directory, so getting this data is dependent upon your tool(s) ability to extract data from this area of the file system. For testing purposes I used my Pixel 3 (running Android 10 – February 2020 patch level), which had been rooted using Magisk.  My tests showed DPS will not be dumped in an Android backup.  Figure 2 shows DPS.

Figure 2
Figure 2.  DPS in the file system:  com.google.android.as

The DPS database of interest here is reflection_gel_events.db in the ./databases folder. This database only has three tables, but the table tracking activity is reflection_event. See Figure 3.

Figure 3
Figure 3.  reflection_event table.

This table is straight forward. Each row represents an event on the device. As seen in Figure 3 there is a column for time and id.  Figure 3 shows an excerpt of the overall activity in the database, and things were chronologically ordered with no intervention needed. Here, you can see I was in the Digital Wellbeing menu within Settings, went back to the top level of the Settings menu, went to the home screen, opened Gmail, and went back to the home screen, and then manually locked the phone (device locks/unlocks are not tracked in this database). There is a small break in the action, but then I unlocked the phone which brought me to the home screen , opened Apple Music for the first time, signed in, went back to the home screen, and then connected the phone to my car for the first time (Android Auto).

Figure 4 shows the rest of the columns.

Figure 4
Figure 4.  The rest of the reflection_event table.

If you read the Digital Wellbeing post, the data in the id column will look familiar. The data in this column documents the apps used via their respective package/subpackage names. Unlike Digital Wellbeing, however, the associated times (in Unix Epoch) are in the same table, so no SQLite query is needed.

The things that initially caught my attention about reflection_gel_events.db were the latLong and cartesian_point columns (who doesn’t like linking activity with location). When I initially found this database, those columns were empty. I tried to populate the columns by using the Pixel 3 over a few days, and they still didn’t populate; thus, my decision to drive Android for a month. Unfortunately, I was never able to get any data to populate in the latLong column, and only a handful of data to populate in the cartesian_point column. I started this initiative right before a lockdown was established in my home state, so the lack of travel may have contributed to the lack of data generation.

The proto column contains a lot of data. As the name suggests, this column contains protobuf data…because, Google. Figure 5 shows a blob in the table and Figure 6 shows the decoded output using protoc.

Figure 5
Figure 5.  BLOB data.
Figure 6
Figure 6.  Decoded protobuf data.

The protobuf data repeats all the data for the entry, with field tags for the event package/subpackage/dependency (Figure 6 – red box) and the event time (Figure 6 – green box).

Here Comes the GEL

As expected, the protobuf data does not look the same each and every time. When I examined some of the protobuf data I found quite a few references to “GEL.” Let’s see Figure 6 again, but this time with a blue box for GEL. See Figure 7.

Figure 7
Figure 7.  The “GEL”

I eventually discovered GEL refers to the Google Experience Launcher, which was a launcher released by Google back in 2013 in Android KitKat (4.4). GEL was re-branded to Google Now Launcher sometime in 2014 and was eventually retired in May of 2018. Basically, GEL refers to Android’s home screen, app drawer, and it is what allows a user to swipe left from the main home screen to get the Google feed. While GEL has been officially gone for two years, some relics still remain as is evidenced here.

Knowing this, I filtered the refection_event table by proto cells that had “GEL” in it I got some interesting results. See Figure 8.

Figure 8
Figure 8.  Filtered by GEL.

When I went back and checked my notes I found that each entry in Figure 8 represents a time when I opened an app by pressing on the icon on the home screen. By doing this, an examiner can bookend app usage by looking for a GEL entry for a particular app and examining the following data until the next GEL entry in the table.  There may be times, however, that a user will utilize multi-tasking to swipe from one app to another without going to the home screen.  The database reflects this, too,  as an examiner would see the package/subpackage for one app and then see the package/subpackage for another with no GEL event.

A Few Differences

DPS contains a lot of the same type of information as Digital Wellbeing but there are some differences. The first notable difference between DPS and Digital Wellbeing is there is no on-device user interface for DPS, but a user can see some of what data is present by checking out myactivity.google.com, which is a user’s dashboard to see what data Google is collecting on them. This allows the user to manage their data, and probably goes towards Google complying with data privacy laws in various jurisdictions. Interestingly enough, this data is included in Google Takeout (hint hint for those of you who have legal processes at your disposal). Figures 9 and 10 show an entry in DPS and a corresponding entry in the Takeout data, respectively. Note the times are not an exact match.

Figure 9
Figure 9.  DPS entry with timestamp 2020-03-17 at 18:06:50 (UTC -0400)
Figure 10
Figure 10.  Corresponding Takout entry.

The good news here is that Figure 10 came from my Google account well after I had deleted WhatsApp. Overall, though, the data in Takeout is a bit less granular than DPS, probably because Google may not want a true “duplicate” of data as it could quickly become unwieldy, and they are not interested in every-single-time you open Twitter to check your feed or what sub-packages were needed for you to do so.

As mentioned earlier, a user has the ability to delete DPS data on the device. I was not able to test how this affects the Google Takeout data for the account, but just be aware there is a correlation between the two, so there may be some effect on Takeout data if a user deletes data from the device.

The lack of granularity can also be said about DPS as compared to Digital Wellbeing, which is the second noticeable difference. DPS does not contain device unlocks, startups, or shutdowns. Comparing the databases for DPS and Digital Wellbeing on my test device found the latter had approximately 19,681 more entries. A significant difference, but because the purpose of each feature is different, this is not surprising. For example, Digital Wellbeing is interested in how many times a user unlocks their phone during the day, and DPS is not.  Examiners do not (and should not) look at an individual data point in a contextual vacuum, including DPS. So, as an example, combining DPS with Digital Wellbeing and Usage Stats could paint a solid picture of user activity on a device.

Another difference is that while the com.google.android.as folder may be present on a device, the reflection_gel_events.db file may be absent. My Android 9 image is Exhibit A. Figures 11 and 12 show the contents of the /data/data/com.google.android.as/databases folder in Android 9 and 10, respectively.

Figure 11
Figure 11.  Android 9 image (no reflection_gel_events.db here).
Figure 12
Figure 12.  Android 10 image (reflection_gel_events.db plus many more).

Google announced App Actions at I/O in May of 2019, which was a month after my Android 9 image, so the lack of reflection_gel_events.db makes sense. It also supports a theory that this database is a component of the on-edge machine learning Google was showcasing. While looking through the com.google.android.as/files folder I found files that may be part of the on-device machine learning models. See Figure 13.

Figure 13
Figure 13.  Machine learning components.

Some of these components probably make the Live Captions and Smart Reply features run, and may also eventually include other features that make Android “smarter.”

The BIG Difference

The third difference is a biggie. In Digital Wellbeing, when an app was deleted entries about it were deleted, too. That is NOT the case with DPS. After generating data for a month, I deleted WhatsApp. Figure 14 shows the same database entry as seen in Figure 9, but after WhatsApp had been deleted from the phone (red box).

Figure 14
Figure 14.  The DPS entry for WhatsApp after deletion.

The timestamps are still present, but the data in the id column is not helpful beyond indicating something was deleted. Here is where the protobuf data comes to the rescue. See Figures 15 and 16.

Figure 15
Figure 15.  Protobuf data from the proto column for WhatsApp after deletion.
Figure 16
Figure 16.  Deoded protobuf data from the entry.

If you recall, the protobuf data for each entry repeats the data for that entry, including the package/subpackage names. This is true even after an app is deleted. So, even though the entry in the id column is gone, an examiner can still determine which app made the entry by looking in the proto column.  Figure 16 clearly shows this entry was for WhatsApp (red box).

A Note

While researching DPS I discovered that just because an Android phone is running Android 10, DPS may be entirely absent.  See Figure 17.

Figure 17
Figure 17.  No DPS here.

Figure 17 is part of a directory listing from a rooted Samsung Galaxy A30, which is running Android 10 with the April 2020 patch. This phone was released in the spring of 2019 with Android 9 and was upgraded to Android 10, and DPS is nowhere to be found. This year’s Samsung flagship, the Galaxy S20, however, does have DPS. I also found information online that indicated the OnePlus 7T and 8 also had DPS (Live Caption), although there was some confusion about whether or not it was a full-fledged version. The Galaxy S20 and OnePlus 8 both have the Snapdragon 865 which has dedicated AI “stuff” (an AI-core or a repurposed DSP core depending on who you ask), and the 7T has the Snapdragon 855+ that also has some enhanced AI “stuff” (an engine). Regardless, it looks as if DPS may be dependent on hardware, which could explain why the Galaxy A30 phone did not have it (the A30 has the Exynos 7904, a “mid-range” chip at best). Regardless, I expect DPS to be in more phones in the future beyond the Pixel series, S20, and the OnePlus 7T and 8. Especially as Google continues their push towards appearing more privacy-focused and phone hardware continues to improve.

Conclusion

While not present on many devices at the moment, I expect DPS to make its way into many more devices in the future. When it does, examiners will have a powerful tool to timeline user activity. Not only does it keep information on currently installed apps, but it also retains data on deleted apps. As it stands now, users can use DPS to track user activity, but it is my hope that it could eventually be used to track activity and location.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s