Two Snaps and a Twist – An In-Depth (and Updated) Look at Snapchat on Android

 

There is an update to this post. It can be found after the ‘Conclusion’ section.

I was recently tasked with examining a two-year old Android-based phone which required an in-depth look at Snapchat. One of the things that I found most striking (and frustrating) during this examination was the lack of a modern, in-depth analysis of the Android version of the application beyond the tcspahn.db file, which, by the way, doesn’t exist anymore, and the /cache folder, which isn’t really used anymore (as far as I can tell). I found a bunch of things that discussed decoding encrypted media files, but this information was years old (Snapchat 5.x). I own the second edition of Learning Android Forensics by Skulkin, Tyndall, and Tamma, and while this book is great, I couldn’t find where they listed the version of Snapchat they examined or the version of Android they were using; what I found during my research for this post did not really match what was written in their book. A lot of things have changed.

Googling didn’t seem to help either; I just kept unearthing the older research. The closest I got was a great blog post by John Walther that examined Snapchat 10.4.0.54 on Android Marshmallow. Some of John’s post lined up with what I was seeing, while other parts did not.

WHAT’S THE BIG DEAL?

Snapchat averages 190 million users daily, which is just under half of the U.S. population, and those 190 million people send three billion snaps (pictures/videos) daily. Personally, I have the app installed on my phone, but it rarely sees any usage. Most of the time I use it on my kid, who likes the filters that alter his voice or requires that he stick out his tongue. He is particularly fond of the recent hot dog filter.

One of the appealing things about Snapchat is that direct messages (DMs) and snaps disappear after a they’re opened. While the app can certainly be used to send silly, ephemeral pictures or videos, some people find a way to twist the app for their own nefarious purposes.

There has been plenty written in the past about how some traces of activity are actually recoverable, but, again, nothing recent. I was surprised to find that there was actually more activity-related data left behind than I thought.

Before we get started just a few things to note (as usual). First, my test data was generated using a Pixel 3 running Android 9.0 (Pie) with a patch level of February 2019. Second, the version of Snapchat I tested is 10.57.0.0, which was the most current version as of 05/22/2019. Third, while the phone was not rooted, it did have TWRP, version 3.3.0-0, installed. Extracting the data was straight forward as I had the Android SDK Platform tools installed on my laptop. I booted into TWRP and then ran the following from the command line:

adb pull /data/data/com.snapchat.android

That’s it. The pull command dropped the entire folder in the same path as where the platform tools resided.

As part of this testing, I extracted the com.snapchat.android folder five different times over a period of 8 days as I wanted to see what stuck around versus what did not. I believe it is also important to understand the volatility of the data that is provided in this app. I think understanding the volatility will help investigators in the field and examiners understand exactly how much time, if any, they have before the data they are seeking is no longer available.

I will add that I tested two tools to see what they could extract: Axiom (version 3.0) and Cellebrite (UFED 4PC 7.18 and Physical Analyzer 7.19). Both tools failed to extract (parsing not included) any Snapchat data. I am not sure if this is a symptom of these tools (I hope not) or my phone. Regardless, both tools extracted nothing.

TWO SNAPS AND…SOME CHANGE

So, what’s changed? Quite a bit as far as I can tell. The storage location of where some of the data that we typically seek has changed. There are enough changes that I will not cover every single file/folder in Snapchat. I will just focus on those things that I think may be important for examiners and/or investigators.

One thing has not changed: the timestamp format. Unless otherwise noted, all timestamps discussed are in Unix Epoch.

The first thing I noticed is that the root level has some new additions (along with some familiar faces). The folders that appear to be new are “app_textures”, “lib”, and “no_backup.” See Figure 1.

Figure 1. Root level of the com.snapchat.android folder.

The first folder that may be of interest is one that has been of interest to forensicators and investigators since the beginning: “databases.” The first database of interest is “main.db.” This database replaces tcspahn.db as it now contains a majority of user data (again, tcspahn.db does not exist anymore). There is quite a bit in here, but I will highlight a few tables. The first table is “Feed.” See Figure 2.

Figure 2. The Feed.

This table contains the last action taken in the app. Specifically, the parties involved in that action (seen in Figure 2), what the action was, and when the action was taken (Figure 3). In Figure 4 you can even see which party did what. The column “lastReadTimestamp” is the absolute last action, and the column “lastReader” show who did that action. In this instance, I had sent a chat message from Fake Account 1 (“thisisdfir”) to Fake Account 2 (“hickdawg957”) and had taken a screenshot of the conversation using Fake Account 1. Fake Account 2 then opened the message.

Enter aFigure 3. Last action. caption

Figure 4. Who did what?
The second table is “Friend.” This table contains anyone who may be my friend. The table contains the other party’s username, user ID, display name, the date/time I added that person as a friend (column “addedTimestamp”), and the date/time the other person added me as a friend (column “reverseAddedTimestamp”). Also seen is any emojis that may be assigned to my friends. See Figures 5, 6, and 7.

Figure 5. Username, User ID, & Display Name.
Figure 6. Friendmojis (Emojis added to my Friends.

Figure 7. Timestamps for when I added friends and when they added me.

Note that the timestamps are for when I originally added the friend/the friend added me. The timestamps here translate back to dates in November of 2018, which is when I originally created the accounts during the creation of my Android Nougat image.

One additional note here. Since everyone is friends with the “Team Snapchat” account, the value for that entry in the “addedTimestamp” column is a good indicator of when the account you’re examining was created.

The next table is a biggie: Messages. I will say that I had some difficulty actually capturing data in this table. The first two attempts involved sending a few messages back and forth, letting the phone sit for a 10 or so minutes, and then extracting the data. In each of those instances, absolutely NO data was left behind in this table.

In order to actually capture the data, I had to leave the phone plugged in to the laptop, send some messages, screenshot the conversation quickly, and then boot into TWRP, which all happened in under two minutes time. If Snapchat is deleting the messages from this table that quickly, they will be extremely hard to capture in the future.

Figure 8 is a screenshot of my conversation (all occurred on 05/30/2019) taken with Fake Account 1 (on the test phone) and Figure 9 shows the table entries. The messages on 05/30/2019 start on Row 6.

Figure 8. A screenshot of the conversation.

Figure 9. Table entries of the conversation.

The columns “timestamp” and “seenTimestamp” are self-explanatory. The column “senderId” is the “id” column from the Friends table. Fake Account 1 (thisisdfir) is senderId 2 and Fake Account 2 (hickdawg957) is senderId 1. The column “feedRowId” tells you who the conversation participants are (beyond the sender). The values link back to the “id” column in the Feed table previously discussed. In this instance, the participants in the conversation are hickdawg957 and thisisdifr.

In case you missed it, Figure 8 actually has two saved messages between these two accounts from December of 2018. Information about those saved messages appear in Rows 1 and 2 in the table. Again, these are relics from previous activity and were not generated during this testing. This is an interesting find as I had completely wiped and reinstalled Android multiple times on this device since the those messages were sent, which leads me to speculate these messages may be saved server-side.

In Figure 10, the “type” column is seen. This column shows the type of message was transmitted. There are three “snap” entries here, but, based on the timestamps, these are not snaps that I sent or received during this testing.

Figure 10. The “types” of messages.
After the “type” column there is a lot of NULL values in a bunch of columns, but you eventually get to the message content, which is seen in Figure 11. Message content is stored as blob data. You’ll also notice there is a column “savedStates.” I am not sure exactly what the entries in the cells are referring to, but they line up with the saved messages.

Figure 11. Message (blob) content.

In Figure 12, I bring up one of the messages that I recently sent.

Figure 12. A sample message.

The next table is “Snaps.” This table is volatile, to say the least. The first data extraction I performed was on 05/22/2019 around 19:00. However, I took multiple pictures and sent multiple snaps on 05/21/2019 around lunch time and the following morning on 05/22/2019. Overall, I sent eight snaps (pictures only) during this time. Figure 13. Shows what I captured during my first data extraction.

Figure 13. I appear to be messing some snaps.
Of the eight snaps that I sent, only six appear in the table. The first two entries in the table pre-date when I started the testing (on 05/21/2019), so those entries are out (they came from Team Snapchat). The first timestamp is from the first snap I sent on 05/22/2019 at 08:24. The two snaps from 05/21/2019 are not here. So, within 24 hours, the data about those snaps had been purged.

On 05/25/2019 I conducted another data extraction after having received a snap and sending two snaps. Figure 14 shows the results.

Figure 14. A day’s worth of snaps.
The entries seen in Figure 13 (save the first two) are gone, but there are two entries there for the snaps I sent. However, there is no entry for the snap I received. I checked all of the tables and there was nothing. I received the snap at 15:18 that day, and performed the extraction at 15:51. Now, I don’t know for sure that a received snap would have been logged. I am sure, however, that it was not there. There may be more testing needed here.

Figure 15 shows the next table, “SendToLastSnapRecipients.” This table shows the user ID of the person I last sent a snap to in the “key” column, and the time at which I sent said snap.

Figure 15. The last snap recipient.

MEMORIES

During the entire testing period I took a total of 13 pictures. Of those 13, I saved 10 of them to “Memories.” Memories is Snapchat’s internal gallery, separate from the phone’s Photos app. After taking a picture and creating an overlay (if desired), you can choose to save the picture, which places it in Memories. If you were to decide to save the picture to your Photos app, Snapchat will allow you to export a copy of the picture (or video).

And here is a plus for examiners/investigators: items placed in Memories are stored server-side. I tested this by signing into Fake Account 1 from an iOS device, and guess what…all of the items I placed in Memories on the Pixel 3 appeared on the iOS device.

Memories can be accessed by swiping up from the bottom of the screen. Figure 16 shows the Snapchat screen after having taken a photo but before snapping (sending) it. Pressing the area in the blue box (bottom left) saves the photo (or video) to Memories. The area in the red box (upper right) are the overlay tools.

Figure 16. The Snapchat screen.

Figure 17 shows the pictures I have in my Memories. Notice that there are only 9 pictures (not 10). More on that in a moment.

Figure 17. My memories. It looks like I am short one picture.

The database memories.db stores relevant information about files that have been saved to Memories. The first table of interest is “memories_entry.” This table contains an “id,” the “snap_id,” and the date the snap was created. There are two columns regarding the time: “created_time” and “latest_created_time.” In Figure 18 there is a few seconds difference between the values in some cells in the two columns, but there are also a few that are the same value. In the cells where there are differences, the differences are negligible.

There is also a column titled “is_private” (seen in Figure 19). This column refers to the My Eyes Only (MEO) feature, which I will discuss shortly. For now, just know that the value of 1 indicates “yes.”

Figure 18. Memories entries.

Figure 19. My Eyes Only status.

(FOR) MY EYES ONLY

I have been seeing a lot of listserv inquires as of late regarding MEO. Cellebrite recently added support for MEO file recovery in Android as of Physical Analyzer 7.19 (iOS to follow), and, after digging around in the memories database, I can see why this would be an issue.

MEO allows a user to protect pictures or videos with a passcode; this passcode is separate from the user’s password for their Snapchat account. A user can opt to use a 4-digit passcode, or a custom alphanumeric passcode. Once a user indicates they want to place a media file in MEO, that file is moved out of the Memories area into MEO (it isn’t copied to MEO).

MEO is basically a private part of Memories. So, just like everything else in Memories, MEO items are also stored server-side. I confirmed this when I signed in to Fake Account 1 from the iOS device; the picture I saved to MEO on the Pixel 3 appeared in MEO on the iOS device. The passcode was the same, too. Snapchat says if a user forgets the passcode to MEO, they cannot help recover it. I’m not sure how true that is, but who knows.

If you recall, I placed 10 pictures in Memories, but Figure 17 only showed 9 pictures. That is because I moved one picture to MEO. Figure 20 shows my MEO gallery.

Figure 20. MEO gallery.

In the memories database, the table “memories_meo_confidential” contains entries about files that have been placed in MEO. See Figure 21.

Figure 21. MEO table in the memories database.

This table contains a “user_id,” the hashed passcode, a “master_key,” and the initialization vector (“iv”). The “master_key” and “initialization vector” are both stored in base64. And, the passcode….well, it has been hashed using bcrypt (ugh). I will add that Cellebrite reports Physical Analyzer 7.19 does have support for accessing MEO files, and, while I did have access to 7.19, I was not able to tell if it was able to access my MEO file since it failed to extract any Snapchat data.

The “user_id” is interesting: “dummy.” I have no idea what that is referring to, and I could not find it anywhere else in the data I extracted.

The next table is “memories_media.” This table. Does have a few tidbits of interesting data: another “id,” the size of the file (“size”), and what type of file (“format”). Since all of my Memories are pictures, all of the cells show “image_jpeg.” See Figures 22 and 23.

Figure 22. “memories_media.”

Figure 23. “memories_media,” part 2.

The next table is “memories_snap.” This table has a lot of information in about my pictures, and brings together data from the other tables in this database. Figure 24 shows a column “media_id,” which corresponds to the “id” in the “memories_media” table discussed earlier. There is also a “creation_time” and “time_zone_id” column. See Figure 24.

Figure 24. id, media_id, creation_time, and time zone.

Figure 25 shows the width and height of the pictures. Also note the column “duration.” The value is 3.0 for each picture. I would be willing to be that number could be higher or lower if the media were videos.

Figure 25 also shows the “memories_entry_id,” which corresponds to the “id” column in the “memories_entry” table. There is also a column for “has_location.” Each of the pictures I placed in Memories has location data associated with it (more on that in a moment).

Figure 25. Picture size, another id, and a location indicator.

Figure 26 is interesting as I have not been able to find the values in the “external_id” or “copy_from_snap_id” columns anywhere.

Figure 26. No clue here.

The data seen in Figure 27 could be very helpful in situations where an examiner/investigator thinks there may be multiple devices in play. The column “snap_create_user_agent” contains information on what version of Snapchat created the the snap, along with the Android version and, in my case, my phone model.

Figure 27. Very helpful.

The column “snap_capture_time” is the time I originally took the picture and not the time I sent the snap.

Figure 28 shows information about the thumbnail associated with each entry.

Figure 28. Thumbnail information.

Figure 29 is just like Figure 27 in its level of value. It contains latitude and longitude of the device when the picture was taken. I plotted each of these entries and I will say that the coordinates are accurate +/- 10 feet. I know the GPS capabilities of every device is different, so just be aware that your mileage may vary.

Figure 29. GPS coordinates!!

Figure 29 also has the column “overlay_size.” This is a good indication if a user has placed an overlay in the picture/video. Overlays are things that are placed in a photo/video after it has been captured. Figure 30 shows an example of an overlay (in the red box). The overlay here is caption text.

Figure 30. An overlay example.

If the value in the overlay_size column is NULL that is a good indication that no overlay was created.

Figure 31 shows the “media_key” and “media_iv,” both of which are in base64. Figure 32 shows the “encrypted_media_key” and “encrypted_media_iv” values. As you can see there is only one entry that has values for these columns; that entry is the picture I placed in MEO.

Figure 31. More base64.

Figure 32. Encrypted stuff.

The next table that may be of interest is “memories_remote_operation.” This shows all of the activity taken within Memories. In the “operation” column, you can see where I added the 10 pictures to Memories (ADD_SNAP_ENTRY_OPERATION). The 11th entry, “UPDATE_PRIVATE_ENTRY_OPERATION,” is where I moved a picture into MEO. See Figure 33.

Figure 33. Remote operations.

The column “serialized_operation” stores information about the operation that was performed. The data appears to be stored in JSON format. The cell contains a lot of the same data that was seen in the “memories_snap” table. I won’t expand it here, but DB Browser for SQLite does a good job of presenting it.

Figure 34 shows a better view of the column plus the “created_timestamp” column. This is the time of when the operation in the entry was performed.

Figure 34. JSON and a timestamp for the operation.

Figure 35 contains the “target_entry” column. The values in these columns refer to the “id”column in the “memories_entry” table.

Figure 35. Operation targets.

To understand the next database, journal, I first have to explain some additional file structure of the com.snapchat.android folder. If you recall all the way back to Figure 1, there was a folder labeled “files.” Entering that folder reveals the folders seen in Figure 36. Figure 37 shows the contents of the “file_manager” folder.

Figure 36. “Files” structure.

Figure 37. file_manager.

The first folder of interest here is “media_package_thumb,” the contents of which can be seen in Figure 38.

Figure 38. Thumbnails?

Examining the first file here in hex finds a familiar header: 0xFF D8 FF E0…yoya. These things are actually JPEGs. So, I opened a command line in the folder, typed ren *.* *.jpg and BAM: pictures! See Figure 39.

Figure 39. Pictures!

Notice there are a few duplications here. However, there are some pictures here that were not saved to memories and were not saved anywhere else. As an example, see the picture in Figure 40.

Figure 40. A non-saved, non-screenshot picture.
Figure 40 is a picture of the front of my employer’s building. For documentation purposes, I put a text overlay in the picture with the date/time I took it (to accompany my notes). I then snapped this picture to Fake Account 2, but did not save it to Memories, did not save it to my Photos app, and did not screenshot it. However, here it is, complete with the overlay. Now, while this isn’t the original picture (it is a thumbnail) it can still be very useful; one would need to examine the “snap” table in the main database to see if there was any activity around the MAC times for the thumbnail.

The next folder of interest is the “memories_media” folder. See Figure 41.

Figure 41. Hmm…

There are 10 items here. These are also JPEGs. I performed the same operation here as I did in the “media_package_thumb” folder and got the results seen in Figure 42.

Figure 42. My Memories, sans overlays.

These are the photographs I placed in Memories, but the caption overlays are missing. The picture that is MEO is also here (the file staring with F5FC6BB…). Additionally, these are high resolution pictures.

You may be asking yourself “What happened to the caption overlays?” I’m glad you asked. They are stored in the “memories_overlay” folder. See Figure 43.

Figure 43. My caption overlays.

Just like the previous two folders, these are actually JPEGs. I performed the rename function, and got the results seen in Figure 44. Figure 45 shows the overlay previously seen in Figure 30.

Figure 44. Overlays.

Figure 45. The Megaman overlay from Figure 30.

The folder “memories_thumbnail” is the same as the others, except it contains just the files in Memories (with the overlays). For brevity’s sake, I will just say the methodology to get the pictures to render is the same as before. Just be aware that while I just have pictures in my Memories, a user could put videos in there, too, so you could have a mixture of media. If you do a mass-renaming, and a file does not render, the file extension is probably wrong, so adjust the file extension(s) accordingly.

Now that we have discussed those file folders, let’s get back to the journal database. This database keeps track of everything in the “file_manager” directory, including those things we just discussed. Figure 46 shows the top level of the database’s entries.

Figure 46. First entries in the journal database.

If I filter the “key” column using the term “package” from the “media_package_thumb” folder (the “media_package_thumb.0” files) I get the results seen in Figure 47.

Figure 47. Filtered results.

The values in the “key” column are the file names for the 21 files seen in Figure 38. The values seen in the “last_update_time” column are the timestamps for when I took the pictures. This is a method by which examiners/investigators could potentially recover snaps that have been deleted.

WHAT ELSE IS THERE?

As it turns out, there are a few more, non-database artifacts left behind which are located in the “shared_prefs” folder seen in Figure 1. The contents can be seen in Figure 48.

Figure 48. shared_prefs contents.

The first file is identity_persistent_store.xml seen in Figure 49. The file contains the timestamp for when Snapchat was installed on the device (INSTALL_ON_DEVICE_TIMESTAMP), when the first logon occurred on the device (FIRST_LOGGED_IN_ON_DEVICE_TIMESTAMP), and the last user to logon to the device (LAST_LOGGED_IN_USERNAME).

Figure 49. identity_persistent_store.xml.

Figure 50. shows the file LoginSignupStore.xml. it contains the username that is logged in.

Figure 50. Who is logged in?

The file user_session_shared_pref.xml has quite a bit of account data in it, and is seen in Figure 51. For starters, it contains the display name (key_display_name), the username (key_username), and the phone number associated with the account (key_phone).

The value “key_created_timestamp” is notable. This time stamp converts to November 29, 2018 at 15:13:34 (EST). Based on my notes from my Nougat image, this was around the time I established Fake Account 1, which was used in the creation of the Nougat image. This might be a good indicator of when the account was established, although, you could always get that data from serving Snapchat with legal process.

Rounding it out is the “key_user_id” (seen in the Friends table of the main database) and the email associated with the account (key_email).

Figure 51. user_session_shared_pref.xml

CONCLUSION

Snapchat’s reputation proceeds it very well. I have been in a few situations where examiners/investigators automatically threw up their hands and gave up after having been told that potential evidence was generated/contained in Snapchat. They wouldn’t even try. I will say that while I always have (and will) try to examine anything regardless of what the general concensus is, I did share a bit of others’ skepticism about the ability to recover much data from Snapchat. However, this exercise has shown me that there is plenty of useful data left behind by Snapchat that can give a good look into its usage.

Update

Alexis Brignoni over at Initialization Vectors noticed that I failed to address something in this post. First, thanks to him for reading and contacting me. 🙂 Second, he noticed that I did not address Cellebrite Physical Analyzer’s (v 7.19) and Axiom’s (v 3.0) ability to parse my test Snapchat data (I addressed the extraction portion only).

We both ran the test data against both tools and found both failed to parse any of the databases. Testing found that while Cellebrite found the pictures I describe in this post, it did not apply the correct MAC times to them (from the journal.db). Axiom failed to parse the databases and failed to identify any of the pictures.

This is not in any way shape or form a knock on or an attempt to single out these two tools; these are just the tools to which I happen to have access. These tools work, and I use them regularly. The vendors do a great job keeping up with the latest developments in both the apps and the operating systems. Sometimes, though, app developers will make a hard turn all of a sudden, and it does take time for the vendors to update their tools. Doing so requires R&D and quality control via testing, which can take a while depending on the complexity of the update.

However, this exercise does bring to light an important lesson in our discipline, one that bears repeating: test and know the limitations of your tools. Knowing the limitations allows you to know when you may be missing data/getting errant readings. Being able to compensate for any shortcomings and manually examine the data is a necessary skillset in our discipline.

Thank you Alexis for the catch and assist!

Google Search Bar & Search Term History – Are You Finding Everything?

Search history.  It is an excellent way to peer into someone’s mind and see what they are thinking at a particular moment in time.  In a court room, search history can be used to show intent (mens rea).  There are plenty of examples where search history has been used in court to establish a defendant’s intent.  Probably the most gruesome was the New York City Cannibal Cop trial, where prosecutors used the accused’s search history against him.  Of course, there is a fine line between intent and protected speech under the First Amendment.

Over the past month and a half I have published a couple of blog posts dealing with Google Assistant and some of the artifacts it leaves behind, which you can find here and here.  While poking around I found additional artifacts present in the same area that have nothing to do with Google Assistant:  search terms.

While I wasn’t surprised, I was; after all, the folder where this data was found had “search” in the title (com.google.android.googlequicksearchbox).  The surprising thing about these search terms is that they are unique to this particular area in Android; they do not appear anywhere else, so it is possible that you or I (or both) could have been missing pertinent artifacts in our examinations (I have missed something).  Conducting a search via this method can trigger Google Chrome to go to a particular location on the Internet, but the term used to conduct the search is missing from the usual spot in the History.db file in Chrome.

My background research on the Google Search Bar (as it is now known) found that this feature may not be used as much as, say, the search/URL bar inside Chrome.  In fact, there are numerous tutorials online that show a user how to remove the Google Search Bar from Android’s Home Screen, presumably to make more space for home screen icons.  I will say, however, that while creating two Android images (Nougat and Oreo), having that search bar there was handy, so I can’t figure out why people wouldn’t use it more.  But, I digress…

Before I get started there are a few things to note.  First, the data for this post comes from two different flavors of Android:  Nougat (7.1.2) and Oreo (8.1).  The images can be found here and here, respectively.  Second, the device used for each image was the same (LG Nexus 5X), and it was rooted both times using TWRP and Magisk.  Third, I will not provide a file structure breakdown here as I did in the Google Assistant blog posts.  This post will focus on the pertinent contents along with content markers within the binarypb files.  I found the binarypb files related to Google Search Bar activity to contain way more protobuff data than those from Google Assistant, so a file structure breakdown is impractical.

Finally, I thought it might be a good idea to give some historical context about this feature by taking a trip down memory lane.

A Quick Background

Back in 2009 Google introduced what, at the time, it called Quick Search Box for Android for Android 1.6 (Doughnut).  It was designed as a place a user could go to type a word or phrase and search not only the local device but also the Internet.  Developers could adjust their app to expose services and content to Quick Search Box so returned results would include their app.  The neat thing about this feature was that it was contextually/location aware, so, for example, I could type the word “weather” and it would display the weather conditions for my current location.  All of this could occur without the need of another app on the phone (depending on the search).

QSB-Doughnut

Google Quick Search Box – circa 2009.

Searching.png

Showtimes…which one do you want?

Prior to Google Assistant, Quick Search Box had a vocal input feature (the microphone icon) that could execute commands (e.g. call Mike’s mobile) and that was about it.  Compared to today this seems archaic, but, at the time, it was cutting edge.

VocalInput.png

Yes, I’m listening.

Fast forward three years to 2012’s Jelly Bean (4.1).  By that time Quick Search Bar (QSB) had been replaced by Google Now, Google’s search and prediction service.  If we were doing Ancestry.com or 23andMe, Google Now would definitely be a genetic relative of Google Search Bar/Google Assistant.  The resemblance is uncanny.

android_41_jelly_bean_ss_08_verge_300.jpg

Mom, is that you?  Google Now in Jelly Bean

The following year, Kit Kat allowed a device to start listening for the hotword “Ok, Google.”  The next big iteration was Now on Tap in 2015’s Marshmallow (6.x), and, with the arrival of Oreo (8.x) we have what we now know today as Google Assistant and the Google Search Bar (GSB).   Recently in Android Pie (9.x) GSB moved from the top part of the home screen to the bottom.

old-navbar-1080x1920

Google Search Bar/Google Assistant at the bottom in Android Pie (9.x).

As of the Fall of 2018 Nougat and Oreo accounted for over half of the total Android install base.  Since I had access to images of both flavors and conducted research on both, the following discussion covers both.  There were a few differences between the two systems, which I will note, but, overall, there was no major divergence.

To understand where GSB lives and the data available, let’s review…

Review Time

GSB and Google Assistant are roommates in both Nougat and Oreo; they both reside in the /data/data directory in the folder com.google.android.googlequicksearchbox.  See Figure 1.

galisting

Figure 1.  GSB & Google Assistant’s home in Android.

This folder holds data about searches that are done from GSB along with vocal input generated by interacting with Google Assistant.  The folder has the usual suspect folders along with several others.  See Figure 2 for the folder listings.

galisting-infile

Figure 2.  Folder listing inside of the googlequicksearchbox folder.

The folder of interest here is app_session.  This folder has a great deal of data, but just looking at what is here one would not suspect anything.  The folder contains several binarypb files, which are binary protocol buffer files.  These files are Google’s home-grown, XML-ish rival to JSON files.  They contain data that is relevant to how a user interacts with their device via Google Assistant and GSB.    See Figure 3.

Figure 3.PNG

Figure 3.  binarypb file (Nougat).

A good deal of the overall structure of these binarypb files differ from those generated by Google Assistant.  I found the GSB binarypb files not easy to read compared to the Google Assistant files.  However, the concept is similar:  there are markers that allow an examiner to quickly locate and identify the pertinent data.

Down in the Weeds

To start, I chose 18551.binarypb in the Nougat image (7.1.2)This search occurred on 11/30/2018 at 03:55 PM (EST).  The search was conducted while the phone was sitting on my desk in front of me, unlocked and displaying the home screen.  The term I typed in to the GSB was “dfir.”  I was presented with a few choices, and then chose the option that took me to the “AboutDFIR” website via Google Chrome.  The beginning of the file appears in Figure 4.

Figure 4.PNG

Figure 4.  Oh hello!

While not a complete match, this structure is slightly similar to that of the Google Assistant binarypb files.  The big takeaway here is the “search” in the blue box.  This is what this file represents/where the request is coming from.  The BNDLs in the red boxes are familiar to those who have read the Google Assistant posts.  While BNDLs are scattered throughout these files, it is difficult to determine where the individual transactions occur within the binarypb files, thus I will ignore them for the remainder of the post.

Scrolling down a bit finds the first area of interest seen in Figure 5.

Figure 5.PNG

Figure 5.  This looks familar.

In the Google Assistant files, there was an 8-byte string that appeared just before each vocal input.  Here there is a four-byte string (0x40404004 – green box) that appears before the search term (purple box).  Also present is a time stamp in Unix Epoch Time format (red box).  The string, 0x97C3676667010000 is read little endian and converted to decimal.  Here, that value is 1543611335575.

Figure 6.PNG

Figure 6.  The results of the decimal conversion.

This time is the time I conducted the search from GSB on the home screen.

Down further is the area seen in Figure 7.   The bit in the orange box looks like the Java wrappers in the Google Assistant files.  The string webj and.gsa.widget.text* search dfir and.gsa.widget.text has my search term “dfir” wrapped in two strings:  “and.gsa.widget.txt.”  Based on Android naming schemas, I believe this to be “Android Google Search Assistant Widget” with text.  This is speculation on my part as I haven’t been able to find anything that confirms or denies this.

Figure 7.PNG

Figure 7.  More search information.

The 4-byte string (green box), my search term (purple box), and the time stamp (red box) are all here.  Additionally, is the string in the blue box.  The string, a 5-byte string 0xBAF1C8F803, is something seen in Google Assistant files.  In the Google Assistant files, this string appeared just prior to the first vocal input in a binarypb file, regardless of when, chronologically, it occurred during the session (remember, the last thing chronologically in the session was the first thing in those binarypb files).  Here, this string occurs at the second appearance of the search term.

Traveling further, I find the area depicted in Figure 8.  This area of the file is very similar to that of the Google Assistant files.

Figure 8.PNG

Figure 8.  A familar layout.

The 16-byte string ending in 0x12 in the blue box is one that was seen in the Google Assistant files.  In those files I postulated this string marked the end of a vocal transaction.  Here, it appears to be doing the same thing.  Just after that, a BNDL appears, then the 4-byte string in the green box, and finally my “dfir” search term (purple box).  Just below this area, in Figure 9, there is a string “android.search.extra.EVENT_ID” and what appears to be some type of identifier (orange box).  Just below that, is the same time stamp from before (red box).

Figure 9.PNG

Figure 9.  An identifier.

I am showing Figure 10 just to show a similarity between GSB and Google Assistant files.  In Google Assistant, there was a 16-byte string at the end of the file that looked like the one shown in Figure 8, but it ended in 0x18 instead of 0x12.  In GSB files, that string is not present.  Part of it is, but not all of it (see the red box).  What is present is the and.gsa.d.ssc. string (blue box), which was also present in Google Assistant files.

Figure 10.PNG

Figure 10.  The end (?).

The next file I chose was 33572.binarypb.  This search occurred on 12/04/2018 at 08:48 AM (EST).  The search was conducted while the phone was sitting on my desk in front of me, unlocked and displaying the home screen.  The term I typed in to the GSB was “nist cfreds.”  I was presented with a few choices, and then chose the option that took me to NIST’s CFReDS Project website via Google Chrome.  The beginning of the file appears in Figure 11.

Figure 11.PNG

Figure 11.  Looks the same.

This looks just about the same as Figure 4.  As before, the pertinent piece is the “search” in the blue box.  Traveling past a lot of protobuff data, I arrive at the area shown in Figure 12.

Figure 12.PNG

Figure 12.  The same, but not.

Other than the search term (purple box) and time stamp (red box) this looks just like Figure 5.  The time stamp converts to decimal 1543931294855 (Unix Epoch Time).  See Figure 13.

Figure 13.PNG

Figure 13.  Looks right.

As before, this was the time that I had conducted the search in GSB.

Figure 14 recycles what was seen in Figure 7.

Figure 14.PNG

Figure 14.  Same as Figure 7.

Figure 15 is a repeat of what was seen in Figures 8 and 9.

Figure 15.PNG

Figure 15.  Same as Figures 8 & 9.

While I am not showing it here, just know that the end of this file looks the same as the first (seen in Figure 10).

In both instances, after having received a set of results, I chose ones that I knew would trigger Google Chrome, so I thought there would be some traces of my activities there.  I started looking at the History.db file, which shows a great deal of Google Chrome activity.  If you aren’t familiar, you can find it in the data\com.android.chrome\app_chrome\Default folder.  I used ol’ trusty DB Browser for SQLite (version 3.10.1) to view the contents.

As it turns out, I was partially correct.

Figure 16 shows the table “keyword_search_terms” in the History.db file.

Figure 16.PNG

Figure 16.  Something(s) is missing.

This table shows search terms used Google Chrome.  The term shown, “george hw bush,” is one that that I conducted via Chrome on 12/01/2018 at 08:35 AM (EST).  The terms I typed in to GSB to conduct my searches, “dfir” and “nist cfreds,” do not appear.  However, viewing the table “urls,” a table that shows the browsing history for my test Google account, you can see when I went to the AboutDFIR and CFReDS Project websites.  See Figures 17 and 18.

Figure 17

Figure 17.  My visit to About DFIR.

Figure 18.PNG

Figure 18.  My visit to NIST’s CFReDS.

The column “last_visit_time” stores the time of last visit to the site seen in the “url” column.  The times are stored in Google Chrome Time (aka WebKit time), which is a 64-bit value in microseconds since 01/01/1601 at 00:00 (UTC).  Figure 19 shows the time I visited AboutDFIR and Figure 20 shows the time I visited CFReDS.

Figure 19

Figure 19.  Time of my visit to About DFIR.

Figure 20

Figure 20.  Time of my visit to NIST’s CFReDS.

I finished searching the Chrome directory and did not find any traces of the search terms I was looking for, so I went back over to the GSB directory and looked there (other than the binarypb files).  Still nothing.  In fact, I did not find any trace of the search terms other than in the binarypb files.  As a last-ditch effort, I ran a raw keyword search across the entire Nougat image, and still did not find anything.

This could potentially be a problem.  Could it be that we are missing parts of the search history in Android?  The History.db file is a great and easy place to look and I am certain the vendors are parsing that file, but are the tool vendors looking at and parsing the binarypb files, too?

As I previously mentioned, I also had access to an Oreo image, so I loaded that one up and navigated to the com.google.android.googlequicksearchbox\app_session folder.  Figure 21 shows the file listing.

Figure 21.PNG

Figure 21.  File listing for Oreo.

The file I chose here was 26719.binarypb.  This search occurred on 02/02/2019 at 08:48 PM (EST).  The search was conducted while the phone was sitting in front of me, unlocked and displaying the home screen.  The term I typed in to the GSB was “apple macintosh classic.”  I was presented with a few choices but took no action beyond that.  Figure 22 shows the beginning of the file in which the “search” string can be seen in the blue box.

Figure 22.PNG

Figure 22.  Top of the new file.

Figure 23 shows an area just about identical to that seen in Nougat (Figures 5 and 12).  My search term can be seen in the purple box and a time stamp in the red box.  The time stamp converts to decimal 1549158503573 (Unix Epoch Time).  The results can be seen in Figure 24.

Figure 23.PNG

Figure 23.  An old friend.

Figure 24

Figure 24.  Time when I searched for “apple macintosh classic.”

Figure 23 does show a spot where Oreo differs from Nougat.  The 4-byte in the green box that appears just before the search term, 0x50404004, is different.  In Nougat, the first byte is 0x40, and here it is 0x50.  A small change, but a change, nonetheless.

Figure 25 shows a few things that appeared in Nougat (Figures 7 & 14).

Figure 25

Figure 25.  The same as Figures 7 & 14.

As seen, the search term is in the purple box, the search term is wrapped in the orange box, the 4-byte string appears in the green box, and the 5-byte string seen in the Nougat and the Google Assistant files is present (blue box).

Figure 26 shows the same objects as those in the Nougat files (Figures 8, 9, & 15).  The 16-byte string ending in 0x12, the 4-byte string (green box), my search term (purple box), some type of identifier (orange box), and the time stamp (red box).

Figure 26.PNG

Figure 26.  Looks familar…again.

While not depicted in this post, the end of the file looks identical to those seen in the Nougat files.

Just like before, I traveled to the History.db file to look at the “keyword_search_terms” table to see if I could find any artifacts left behind.  See Figure 27.

Figure 27.PNG

Figure 27.  Something is missing…again.

My search term, “apple macintosh classic,” is missing.  Again.  I looked back at the rest of the GSB directory and struck out.  Again.  I then ran a raw keyword search against the entire image.  Nothing.  Again.

Out of curiosity, I decided to try two popular forensic tools to see if they would find these search terms.  The first tool I tried was Cellebrite Physical Analyzer (Version 7.15.1.1).  I ran both images through PA, and the only search terms I saw (in the parsed data area of PA) were the ones that were present in Figures 16 & 27; these terms were pulled from the “keyword_search_terms” table in the History.db file.  I ran a search across both images (from the PA search bar) using the keywords “dfir,”“cfreds,” and “apple macintosh classic.”  The only returned hits were the ones from the “urls” table in the History.db file  of the Nougat image; the search term in the Oreo image (“apple macintosh classic”) did not show up at all.

Next, I tried Internet Evidence Finder (Version 6.23.1.15677).  The Returned Artifacts found the same ones Physical Analyzer did and from the same location but did not find the search terms from GSB.

So, two tools that have a good foot print in the digital forensic community missed my search terms from GSB.  My intentions here are not to to speak ill of either Cellebrite or Magnet Forensics, but to show that our tools may not be getting everything that is available (the vendors can’t research everything).  It is repeated often in our discipline, but it does bear repeating here:  always test your tools.

There is a silver lining here, though.  Just to check, I examined my Google Takeout data, and, as it turns out, these searches were present in what was provided by Google.

Conclusion

Search terms and search history are great evidence.  They provide insight in to a user’s mindset and can be compelling evidence in a court room, civil or criminal.  Google Search Bar provides users a quick and convenient way to conduct searches from their home screen without opening any apps.  These convenient searches can be spontaneous and, thus, dangerous; a user could conduct a search without much thought given to the consequences or how it may look to third parties.  The spontaneity can be very revealing.

Two major/popular forensic tools did not locate the search terms from Google Search Bar, so it is possible examiners are missing search terms/history.  I will be the first to admit, now that I know this, that I have probably missed a search term or two.  If you think a user conducted a search and you’re not seeing the search term(s) in the usual spot, try the area discussed in this post.

And remember:  Always.  Test.  Your.  Tools.

Update

A few days after this blog post was published, I had a chance to test Cellebrite Physical Analyzer, version 7.16.0.93.  This version does parse the .binarypb files, although you will get multiple entries for the same search, and some entries may have different timestamps.  So, caveat emptor; it will be up to you/the investigator/both of you to determine which is accurate.

I also have had some time to discuss this subject further with Phil Moore (This Week in 4n6), who has done a bit of work with protobuf files (Spotify and the KnowledgeC database).  The thought was to use Google’s protoc.exe (found here) to encode the .binarypb files and then try to decode the respective fields.  Theoretically, this would make it slightly easier than manually cruising through the hexadecimal and decoding the time manually.  To test this, I ran the file 26719.binarypb through protoc.exe.  You can see the results for yourself in Figures 28, 29, and 30, with particular attention being paid to Figure 29.

Figure 28

Figure 28. Beginning of protoc output.

 

Figure 29

Figure 29.  Middle part of the protoc output (spaces added for readability).

 

Figure 30

Figure 30.  Footer of the protoc output.

In Figure 28 the “search” string is identified nicely, so a user could easily see that this represents a search, but you can also see there is a bunch of non-sensical data grouped in octets.  These octets represent the data in the .binarypb file, but how it lines up with the hexadecimal values/ASCII values is anyone’s guess.  It is my understanding that there is a bit of educated guessing that occurs when attempting to decode this type of data.  Since protobuf data is serialized and the programmers have carte blanche in determining what key/value pairs exist, the octets could represent anything.

That being said, the lone educated guess I have is that the octet 377 represents 0xFF.  I counted the number of 377’s backwards from the end of the octal time (described below) and found that they matched (24 – there were 24 0xFF’s that proceeded the time stamp seen in Figure 23).  Again, speculation on my part.

Figure 29 is the middle of the output (I added spaces for readability).  The area in the red box, as discovered by Phil, is believed to the be the timestamp, but in an octal (base-8) format…sneaky, Google.  The question mark at the end of the string lines up with the question mark seen at the end of each timestamp seen in the figures of this article.  The area in the green box shows the first half of the Java wrapper that was discussed and seen in Figure 25.  The orange box contains the search string and the last half of the Java wrapper.

Figure 30 shows the end of the protoc output with the and.gsa.d.ssc.16 string.

So, while there is not an open-source method of parsing this data as of this writing, Cellebrite, as previously mentioned, has baked this into the latest version of Physical Analyzer, but care should be taken to determine which timestamp(s) is accurate.

OK Computer…er…Google. Dissecting Google Assistant (Part Deux)

NoDisassemble

In part two of this article I will be looking at Google Assistant artifacts that are generated when using a device outside of the car (non-Android Auto). Since this post is a continuation of the first, I will dispense with the usual pleasantries, and jump right into things.  If you have not read Part 1 of this post (dealing with Google Assistant artifacts generated when using Google Assistant via Android Auto), at least read the last portion, which you can do here.  The data (the phone extraction) discussed in both posts can be found here.  Just know that this part will not be as long as the first, and will, eventually, compare the Google Assistant artifacts generated in Android Auto to those generated just using the device.

If you don’t feel like clicking over, let’s recap:

A Slight Review

Google Assistant resides in the /data/data directory.  The folderis com.google.android.googlequicksearchbox.  See Figure 1.

galisting

Figure 1.  Google Assistant’s home in Android.

This folder also holds data about searches that are done from the Quick Search Box that resides at the top of my home screen (in Oreo).  The folder has the usual suspect folders along with several others.  See Figure 2 for the folder listings.

galisting-infile

Figure 2.  Folder listing inside of the googlequicksearchbox folder.

The folder of interest here is app_session.  This folder has a great deal of data, but just looking at what is here one would not suspect anything.  The folder contains several binarypb files, which I have learned, after having done additional research, are binary protocol buffer files.  These files are Google’s home-grown, XML-ish rival to JSON files.  They contain data that is relevant to how a user interacts with their device via Google Assistant.    See Figure 3.

binarypbs

Figure 3.  binarypb files.

Each binarypb file here represents a “session,” which I define as each time Google Assistant was invoked.  Based on my notes, I know when I summoned Google Assistant, how I summoned it, and what I did when I summoned it.  By comparing my notes to the MAC times associated with each binarypb file I identified the applicable files for actions taken inside of the car (via Android Auto) and those taken outside of the car.

During my examination of the binarypb files that were created during sessions inside of the car, I found similarities between each file, which are as follows:

  1. Each binarypb file will start by telling you where the request is coming from (car_assistant).
  2. What is last chronologically is first in the binarypb Usually, this is Google Assistant’s response (MP3 file) to a vocal input just before being handed off to whatever service (e.g. Maps) you were trying to use.  The timestamp associated with this is also at the beginning of the file.
  3. A session can be broken down in to micro-sessions, which I call vocal transactions.
  4. Vocal transactions have a visible line of demarcation by way of the 16-byte string ending in 0x12.
  5. A BNDL starts a vocal transaction, but also further divides the vocal transaction in to small chunks.
  6. The first vocal input in the binarypb file is marked by a 5-byte string: 0xBAF1C8F803, regardless of when, chronologically, it occurred in the session.
  7. Each vocal input is marked by an 8-byte string:   While the 5-byte string appears at the first in the binarypb file only (along with the 8-byte string), the 8-byte string appears just prior to each and every vocal input in the file.
  8. When Google Assistant doesn’t think it understands you, it generates different variations of what you said…candidates…and then selects the one it thinks you said.
  9. In sessions where Google Assistant needs to keep things tidy, it will assign an identifier. There does not appear to be any consistency (as far as I can tell) as to the format of these identifiers.
  10. The end of the final vocal transaction is marked by a 16-byte string ending in 0x18.

Visually, sessions via Android Auto can be seen in Figure 4, and vocal transactions can be seen in Figure 5.

img_0075

Figure 4.  Visual resprensetation of a session.

 

img_0074

Figure 5.  Visual representation of vocal transactions.

One additional notation here.  I was contacted by a reader via Twitter and asked about adding byte offsets to Figures 4 and 5.  Unfortunately, the byte offsets beyond the header are never consistent.  This is due to requests always being different, and, as a result, Google Assistant’s response (whether vocally, by action, or both) are always different.  I think the thing to keep in mind here is that there is a structure and there are some markers to help examiners locate this data.

A Deep Dive

To start, I chose 13099.binarypb.  This session occurred on 01/28/2019 at 12:41 PM (EST) and involved reading new text messages and dictating a response.  The session was initiated by “Ok, Google” with the phone sitting on my desk in front of me while the phone was unlocked and displaying the home screen.  The session went like this:

First Dialogue

Figure 6 shows the top of the binarypb file.  In the blue box is something familiar:  the 0x155951 hex value at offset 0x10.  This string was also present in the binarypb files generated while inside the car (via Android Auto).  In the orange box “opa” appears.  This string appears at the top of each binarypb file generated as a result of using Google Assistant outside of the car.  I suspect (based on other data seen in these files) that this is a reference to the Opa programming language.  This would make sense as I see references to Java, too, which is used throughout Android.  Additionally, Opa is aimed at both client-side and server-side operations (Node.js on the server and JavaScript on the client side).  Again, this is speculation on my part, but the circumstantial evidence is strong.

Figure 6

Figure 6. Top of 13099.binarypb.

In the red boxes are the oh-so-familiar “BNDL’s.” In the green box the string “com.google.android.googlequicksearchbox” is seen.  This is the folder in which the Quick Search Box resides, along with the session files for Google Assistant.

Just below the area in Figure 6 is the area in Figure 7.  There are a couple of BNDL’s in this area, along with the area in the orange.  This string appears to be indicating this part of the file was caused by a change in the conversation between Google Assistant and myself; “TRIGGERED_BY” and “CONVERSATION_DELTA.” See Figure 7.

Figure 7

Figure 7. A change in conversation triggered this vocal transaction

The area in the blue box is interesting as it is a string that is repeated throughout this session.  I suspect…loosely…this is some type of identifier, and the string below it (in Figure 8) is some type of token.

Figure 8.PNG

Figure 8.  ID with possible token…?

I will stop here for a second.  There was a noticeable absence at the top of this file.  There was no MP3 data here.  A quick scan of this entire file finds no MP3 data at all.  Determining whether this is unique to this particular file or systemic trend will require examining other files (later in this article).

After the area in Figure 8 there was quite a bit of protocol buffer data.  Eventually, I arrived at the area depicted in Figure 9.  In it you can see the identifier from Figure 7 (blue box), a bit more data, and then a time stamp (red box).  The value is 0x65148E9568010000, which, when read little endian is 1548697343077 (Unix Epoch Time).  Figure 10 shows the outcome using DCode.

Figure 9.PNG

Figure 9.  Identifier and Unix Epoch Time time stamp.

 

Figure 10.PNG

Figure 10. Time stamp from Figure 9.

The time stamp here is about a minute ahead of when I initiated the session.  Remember what I said about the last thing chronologically being the first thing in the file?  I suspect the last thing I said to Google Assistant will be the first vocal input data I see.  See Figure 11.

Figure 11.PNG

Figure 11.  Last vocal input of the session.

There is one bit of familiar data in here.  If you read the first part of this article you will know that the string in the blue box (0xBAF1C8F803) appeared just before the first vocal input in a binarypb file, which is usually the last vocal input data of the session.  It did not appear anywhere else within the file.  It appears here, too, in a session outside of the car.

In the orange box is what appears to be some Java data indicating where this session started:  “hotword.”  The hotword is the trigger phrase for Google Assistant, which, for me is “Ok, Google.”  The 8-byte string in the green box (0x010C404000040200) is consistent throughout the file (save one location – discussed later), and, as suspected, my last vocal input that I provided Google Assistant (purple box).  A BNDL appears at the end in the red box.

Figure 12 shows some familiar data (from Figures 7 & 8):  TRIGGERED_BY, CONVERSATION_DELTA, the identifier (blue box) and what I believe to be some token (red box).  Note that the suspected token here matches that seen in Figure 8.

Figure 12

Figure 12.  A rehash of Figures 7 & 8.

 

Figure 13.PNG

Figure 13.  The identifier again and another time stamp.

After some more protocol buffer data I find the area in Figure 13.  It looks the same as the area shown in Figure 9, and the time stamp is the same.

Figure 14 is a somewhat recycled view of what was seen in Figure 11, but with a twist.  The Java data which seems to indicate where the query came from wraps the vocal input (“no”); see the orange box.  A BNDL is also present.

Figure 14.PNG

Figure 14.  Vocal input with a Java wrapper.

Also seen in Figure 14 is another time stamp in the red box.  The value is 0x65148E9568010000, which is decimal 1548697279859.  As before, I used DCode to convert this from Unix Epoch Time to 01/28/2019 at 12:41:23 (EST).  This is the time I originally invoked Google Assistant.

Figure 15 shows some more data, and the end of the vocal transaction (see my Part 1 post).  This is marked by the velvet:query_state:search_result_id string (purple box) and the 16-byte hex value of 0x00000006000000000000000000000012 (orange box).  The string and accompanying hex value are the same ones seen in the binarypb files generated by interaction with Google Assistant via Android Auto.

Figure 15

Figure 15.  Data marking the end of the vocal transaction.

Figure 16 shows the start of a new vocal transaction.  The BNDL (seen at the bottom of Figure 15, but not marked) is in the red box.  Just below it is the 8-byte string in the green box.  Note that the last byte is 0x10 and not 0x00 as seen in Figure 11.  My vocal input appears in the purple box; this input is what started the session.  Just below it is another BNDL.  See Figure 16.

Figure 16.PNG

Figure 16.  New vocal transaction.

The items below the BNDL are interesting.  The orange box is something previously seen in this file:  TRIGGERED_BY.  However, the item in the blue box is new.  The string is QUERY_FROM_HOMESCREEN, which is exactly what the phone was displaying when I invoked Google Assistant.  The phone was on, unlocked, and I used the hotword to invoke Google Assistant, which leads me to the string in the brown box: “INITIAL_QUERY.”  The phrase “read my new text messages” was my original request.  This area seems to imply that my phrase was the initial query and that it was made from the home screen.  Obviously, there is plenty of more testing that needs to be done to confirm this, but it is a good hypothesis.

Figure 17.PNG

Figure 17.  A time stamp and a “provider.”

In Figure 17 there is a time stamp (red box):  the decimal value is 1548697279878 (Unix Epoch Time) and the actual time is 01/28/2019 at 12:41:19 (EST).  Again, this is the time Google Assistant was invoked.  The portion in the blue box, while not a complete match, is data that is similar to data seen in Android Auto.  I highlighted the whole box, but the area of interest is voiceinteraction.hotword.HotwordAudioProvider /34.  In the Android Auto version, the related string was projection.gearhead.provider /mic /mic.  In the Part 1 post, I indicated that the /mic /mic string indicated where the vocal input was coming from (my in-car microphone used via Android Auto).  Here I believe this string indicates the origin of the Google Assistant invocation is via the hotword, although I am not completely sure about the /34.

The area in the blue box in Figure 18 is new.  I have tried to find what the data in the box means or its significance, and I have been unable to do so.  In addition to searching the Google developer forums, I pulled the phone’s properties over ADB in order to see if I could determine if the data was referring to the on-board microphone and speaker (ear piece), but the list of returned items did not have any of this data.  At this point I have no idea what it means.  If someone knows, please contact me and I will add it to this article and give full credit.

Figure 18-1-1

Figure 18.  Something new.

I had to scroll through some more protocol buffer data to arrive at the area in Figure 18-1.  There are several things here:  the velvet:query_state:search_result_id with the accompanying 16-byte string ending in 0x12 (brown boxes), BNDLs (red boxes), the 8-byte string just prior to my vocal input (green box), my vocal input (purple box), the TRIGGERED_BY, CONVERSATION_DELTA strings (orange box – my response “yes” was a result in a change in the conversation), and the identifier that I had seen earlier in the file (blue box).  Note that while the string in the green box matches the string seen in Figure 11, it differs from the one seen in Figure 18-1.  The string in Figure 18-1 ends in 0x10 whereas the string here and Figure 11 both end in 0x00.

Figure 18

Figure 18-1.  The end of one vocal transaction and the beginning of another.

Just past the identifier seen in Figure 18-1, there was another string that I suspect is a token.  This string starts out the same as the one seen in Figures 8 and 12, but it does differ.  See Figure 19.

Figure 19.PNG

Figure 19.  A new “token.”

Scrolling through more protocol buffer data finds the area seen in Figure 20.  Here I find another time stamp (red box).  The decoding methodology is the same as before, and it resulted in a time stamp of 01/28/2019 at 12:41:42 (EST).  This would have been around the time that I indicated that I wanted to reply to the text messages (by saying “yes”) Google Assistant had read to me.  Additionally, the Java string appears (orange box), and the end of the vocal transaction is seen with the velvet:query_state:search_result_id and the accompanying 16-byte string ending in 0x12 (blue boxes).

Figure 20

Figure 20.  The end of another vocal transaction.

Figure 21 has my dictated message in it (purple box), along with some familiar data, and a familiar format.

Figure 21

Figure 21.  A familiar face.

At the top is a BNDL (red box), the 8-byte string ending in 0x00 (green box), another BNDL (red box), the TRIGGERED_BY, CONVERSATION_DELTA strings (orange box), and the identifier again (blue box).  In Figure 22 another “token” is found (red box).  This is the same one as seen in Figure 19.

Figure 22.PNG

Figure 22.  Another “token.”

Yet more protocol buffer data, and yet more scrolling takes me to the area in Figure 23.  In the red box is another time stamp.  In decimal it is 1548697307562 (Unix Epoch Time), which converts to 01/28/2019 at 12:41:47 (EST).  This would have been around the time I dictated my message to Google Assistant.  The identifier also appears at the foot of the protocol buffer data (blue box).

Figure 23.PNG

Figure 23.  Another time stamp.

Figure 24 shows the same data as in Figure 20:  the end of a vocal transaction.  The orange box contains the Java data, and the blue box contains the velvet:query_state:search_result_id and the accompanying 16-byte string ending in 0x12.

Figure 24

Figure 24.  End of another vocal transaction.

Beyond my vocal input (purple box), the area seen in Figure 25 is the same as those seen in Figures 18-1 & 21.  I even marked them the same… BNDL (red box), the 8-byte string ending in 0x00 (green box), another BNDL (red box), the TRIGGERED_BY, CONVERSATION_DELTA strings (orange box), and the identifier again (blue box).

Figure 25

Figure 25.  The top of another vocal transaction.

Figure 26 shows an area after some protocol buffer data that trailed the identifier in Figure 25.  The notable thing here is the time stamp in the red box.  It is decimal 1548697321442 (Unix Epoch Time), which translate to 01/28/2019 at 12:42:01 (EST).  This would have lined up with when I sent the dictated text message.

Figure 26.PNG

Figure 26.  Time stamp from “No.”

Figure 27 shows the end of the vocal transaction here.  In the orange box is the Java data, with the velvet:query_state:search_result_id and the accompanying 16-byte string ending in 0x12 in the blue box.

Figure 27.PNG

Figure 27.  The end of a vocal transaction.

Figure 28 looks just like Figures 18-1, 21 & 25.  The only difference here is my vocal input (“no”).  This was the last thing I said to Google Assistant in this session, so I expect this last portion of the file (save the very end) to look similar to the top of the file.

Figure 28

Figure 28.  Look familiar?

Figure 29 contains a time stamp (red box), which appears after a bit of protocol buffer data.  It is decimal 1548697343077 (Unix Epoch Time), which converts to 12:42:23 (EST).  This is the same time stamp encountered in this session file seen in Figure 9.

Figure 29.PNG

Figure 29.  The last/first time stamp.

Figure 30 shows the end of the session file with the orange box showing the usual Java data.  The end of this file, as it turns out, looks very similar to end of session files generated via Android Auto.  Three things are present here that are also present in the end of the Android Auto session files.  First, the velvet:query_state:search_result_id and the accompanying 16-byte string ending in 0x18 in the blue box.  Second, the 9-byte string, 0x01B29CF4AE04120A10 in the purple box. Third, the string “and.gsa.d.ssc.” is present in the red box.

Figure 30

Figure 30.  A familiar ending.

So, right away I see quite a bit of similarities between this session file and the ones generated by Android Auto.  In order to have some consistency between these files and those from Android Auto, the next file I examined involved me asking for directions to my favorite coffee joint.

The next file I examined was 13128.binarypb.  This session occurred on 01/28/2019 at 12:43 PM (EST) and involved reading new text messages and dictating a response.  The session was initiated by “Ok, Google” with the phone sitting on my desk in front of me, unlocked, and displaying the home screen.  The session went like this:

Second Dialogue

The screen switched over to Google Maps and gave me the route and ETA.  I did not choose anything and exited Maps.

The top of 13128.binarypb looks identical to 13099.binarypb (Figure 6).  See Figure 31.

Figure 31

Figure 31.  A familiar sight.

The gang is all here.  The string 0x155951 (blue box), “opa” (orange box), com.google.android.googlequicksearchbox (green box), and a couple of BNDL’s (red box).

While no data of interest resides here, I am including Figure 32 just to show that the top of 13128 is just like 13099.

Figure 32.PNG

Figure 32.  Nothing to see here.

A quick note here: this file is just like 13099 in that there is no MP3 data at the beginning of the file. As before, I scanned the rest of the file and found no MP3 data at all. So, this is a definite difference between the Android Auto and non-Android Auto session files.

Figure 33 is something I had seen in the previous file (see Figure 16), but further down.  The blue and orange boxes contain the TRIGGERED_BY and QUERY_FROM_HOMESCREEN strings, respectively.  Just like my previous session, this session was started with the phone on, unlocked, and by using the hotword to invoke Google Assistant, which leads me to the string in the red box: “INITIAL_QUERY.”  This area seems to imply that whatever vocal input is about to show up is the phrase that was the initial query and that it was made from the home screen.

Figure 33.PNG

Figure 33.  Query From Home Screen, Triggered By, Launched On, & Initial Inquiry.

Figure 34 looks almost identical to Figure 18-1.  The red box contains a time stamp, which is decimal 1548697419294 (Unix Epoch Time).  When converted it is 01/28/2019 at 12:43:39 (EST).  The blue box contains the string voiceinteraction.hotword.HotwordAudioProvider /49.  The /49 is different than the one seen in Figure 18-1, though (/34).  Again, I am not sure what this is referring to, and I think it warrants more testing.

Figure 34.PNG

Figure 34.  The query source and a time stamp.

Scrolling down just a hair finds the area in Figure 35.  The orange box contains Java data we have seen before but with a small twist.  The string webj and.opa.hotword* search and.opa.hotword, with the twist being “search” in the middle.  As seen in the first file, it’s almost as if the term in the middle is being wrapped (my “no” was in wrapped as seen in Figure 14).

Figure 35.PNG

Figure 35.  Something old and something old.

The area in the red box is the same data seen in Figure 18.

Figure 36 also contains some familiar faces.  My vocal input is in the purple box, the 5-byte blue string that usually appears at the first vocal input of the session, 0xBAF1C8F803, is here.

Figure 36.PNG

Figure 36.  The first vocal input of the session.

An 8-byte string previously seen in 13099 is also here (see Figure 16).  Note that this string ends in 0x10.  In 13099 all of the 8-byte strings, save one, ended in 0x00.  The one that did end in 0x10 appeared with the first vocal input of the session (“read my new text messages”).  Here, we see the string ending in 0x10 with the only vocal input of the session.  I hypothesize that the 0x10 appears before the first vocal input of the session, with any additional vocal input appearing with the 8-byte string ending in 0x00.  More research is needed to confirm, which is beyond the scope of this article.

Figures 37 and 38 shows the same data as seen in Figure 33 and 34.

Figure 37.PNG

Figure 37.  Same ol’ same ol’.

 

Figure 38.PNG

Figure 38.  Same ol’, part deux.

Figure 39 shows the mysterious string with the speaker id (red box) and Figure 40 shows my vocal input inside of a Java wrapper (orange box), which is similar to what was seen in 13099 (Figure 14).

Figure 39

Figure 39.  Speaker identifier?

 

Figure 40.PNG

Figure 40.  A Java wrapper and a time stamp.

The time stamp seen in Figure 40 is the same as the other time stamps seen in this session file except for the first byte.  The other bytes are 0x1E, whereas the byte seen here is 0x08; this causes the decimal value to shift from 1548697419294 to 1548697419272.  Regardless, the time here is the same:  01/28/2019 at 12:43:39 PM (EST).  The millisecond value is different:  294 versus 272, respectively.

Figure 41 shows the end of the vocal transaction, which is marked by the  velvet:query_state:search_result_id and the accompanying 16-byte string ending in 0x12 in the blue box.

Figure 41.PNG

Figure 41.  The end of the vocal transaction.

The start of a new vocal transaction is seen in Figure 42.  The 8-byte value seen in the green box ends with 0x10, which keeps in line with my theory discussed earlier in this article.  My vocal input (the only input of the session) is seen in the purple box.  A BNDL is seen at the start of the transaction (red box) with another one at the end (red box).

Figure 42.PNG

Figure 42.  The start of another vocal transaction.

In the interest of brevity, I will say that the next bit of the session file is composed of what is seen in Figures 37, 38, and 39 (in that order).  The time stamp is even the same as the one seen in Figure 38.  The next area is the last part of the session file as seen in Figure 43.

Figure 46

Figure 43.  The end!

If Figure 43 looks familiar to you, that is because it is.  I color coded the boxes the same way as I did in Figure 31.  Everything that was there is here:   the Java data (orange box), the velvet:query_state:search_result_id and the accompanying 16-byte string ending in 0x18 in the blue box,  the 9-byte string, 0x01B29CF4AE04120A10 in the purple box, and the string “and.gsa.d.ssc.” is present in the red box.

So What Changed…If Anything?

At the beginning of this article I reviewed some consistencies between the Android Auto session files I examined.   After examining he non-Android Auto files, I thought it would be beneficial to revisit those consistencies to see what, if anything changed.  The original statements are in italics, while the status here is just below each item.

  • Each binarypb file will start by telling you where the request is coming from (car_assistant).

This is still correct except “car_assistant” is replaced by “opa” and “googlequicksearchbox.”

  • What is last chronologically is first in the binarypb file. Usually, this is Google Assistant’s response (MP3 file) to a vocal input just before being handed off to whatever service (e.g. Maps) you were trying to use.  The timestamp associated with this is also at the beginning of the file.

This is still correct, minus the part about the MP3 data.

  • A session can be broken down in to micro-sessions, which I call vocal transactions.

This is still correct.

  • Vocal transactions have a visible line of demarcation by way of the 16-byte string ending in 0x12.

This is still correct.

  • A BNDL starts a vocal transaction, but also further divides the vocal transaction in to small chunks.

This is still correct.

  • The first vocal input in the binarypb file is marked by a 5-byte string: 0xBAF1C8F803, regardless of when, chronologically, it occurred in the session.

This is still correct.

  • Each vocal input is marked by an 8-byte string:   While the 5-byte string appears at the first in the binarypb file only (along with the 8-byte string), the 8-byte string appears just prior to each and every vocal input in the file.

Eh…sorta.  While the values in the 8-bytes change between Android Auto and non-Android Auto, there is some consistency within the fact that there is a consistent 8-byte string.  Further, the last byte of the 8-byte string in the non-Android Auto version varies depending on whether or not the vocal input is chronologically the first input of the session.

  • When Google Assistant doesn’t think it understands you, it generates different variations of what you said…candidates…and then selects the one it thinks you said.

Unknown.  Because I was in an environment which was quiet, and I was near the phone, Google Assistant didn’t seem to have any trouble understanding what I was saying.  It would be interesting to see what would happen if I introduced some background noise.

  • In sessions where Google Assistant needs to keep things tidy, it will assign an identifier. There does not appear to be any consistency (as far as I can tell) as to the format of these identifiers.

This is correct.  In the 13099 file, there were multiple things happening, so an identifier with something that resembled a token was present.

  • The end of the final vocal transaction is marked by a 16-byte string ending in 0x18.

Still correct.

For those of you that are visual learners, I am adding some diagrams at the end that shows the overall, generalize structure of both a session and a vocal transaction.  See Figures 44 and 45, respectively.

OutOfCar-SessionFile.JPG

Figure 44.  Session file.

 

OutOfCar-VocalTransaction

Figure 45.  Vocal transaction.

Conclusion

There is way more work to do here in order to really understand Google Assistant.  Phil Moore, of This Week in 4n6 fame, mentioned the Part 1 of this article recently on the This Month in 4N6 podcast, and he made a very accurate statement:  Google Assistant is relatively under researched.  I concur.  When I was researching this project, I found nothing via Google’s developer forums, and very little outside of those forums.  There just isn’t a whole lot of understanding about how Google Assistant behaves and what it leaves behind on a device.

Google Assistant works with any device that is capable of running Lollipop (5.0) or higher; globally, that is a huge install base!  Additionally, Google Assistant can run on iOS, which adds to the install base, and is a whole other batch of research.  Outside of handsets there are the Google Home speakers, on which there has been some research, Android TVs, Google Home hubs, smart watches, and Pixelbooks/Pixel Slates. Google is making a push in the virtual assistant space, and is going all in with Google Assistant (see Duplex). With the all of these devices capable of running Google Assistant it is imperative that practitioners learn how it behaves and what artifacts it leaves behind.

OK Computer…er…Google. Dissecting Google Assistant (Part 1)

A few weeks ago, I posted a blog about some research I conducted on Android Auto, and I mentioned there was some interesting data left behind by Google Assistant when using Android Auto.  Based on what I found there, I decided to go further down the virtual assistant rabbit hole to see what I could find.

As far as virtual assistants go, I use Siri.  When I had a newborn, Siri was used a lot.  In addition to turning on/off lights or playing music, I used Siri to turn on certain appliances (via smart plugs), respond to texts, and make phone calls as my hands were usually holding a baby, trying to do something for/to/with a baby, or I was trying to help my spouse do those things.  Siri was really helpful then.  Siri is still useful, but, nowadays, my primary use of Siri is in the car.  There are times where I still yell at my phone or HomePod to find a TV show, play a song, turn on a light, or answer a quick math question.  For other things such as search and typing a message (outside of the car), I’m old fashioned.

I have been fascinated by talking computers/A.I. for a long time.  According to my parents, I used to love Knight Rider.  It had the Knight Industries Two Thousand (K.I.T.T.) – the snarky, crime-fighting automotive sidekick of the Hoff.  As you can see from the GIF above, I also like Star Trek.  Captain Kirk, Spock, et al. have been verbally interacting with their computers since the 1960’s.  WOPR/Joshua, check.  Project 2501, check.  The Architect, check.  And, the crème de la crème:  HAL 9000, check check check.

While researching Google Assistant, I stumbled across an article that had some interesting statistics.  In 2017, there was a 128.9% year-over -year increase in the use of voice-activated assistants, with an expected growth of 40% in 2019.  Another statistic:  2 out of 5 adults use voice search at least once a day; this was in 2017, so I suspect this number is higher by now.  This explosion, in my opinion, started when Alexa arrived.  Say what you like about Amazon, but they were smart to open Alexa to developers.  Alexa is everywhere and owns around 70% of the virtual assistant market.  With Amazon’s recent acquisition of eero (to my dismay), wireless routers, the televisions of the 21st century, will have Alexa in them.  Alexa. Is. Everywhere.  I am sure Google will follow, so I feel it is important to understand any artifact Google Assistant may leave behind on a device.

There are three things to be aware of before I get started.  First, the data I found resides in the /data/ directory, and, thus, is not easily accessible on newer devices unless the device is rooted, or you have a tool that can access this area.  Second, a warning:  this post is lengthy, but in order to understand the patterns and data in the three files I examine, it is necessary.

Finally, this will be a two-parter.  There was way too much data to cover everything in just one post.  This post will examine the data left behind by Google Assistant when used via Android Auto.  The second post will examine the data left behind by Google Assistant when used outside of the car (i.e. when I yell at the device).

One final note:  this data was generated on a rooted Nexus 5X running Android Oreo (8.1), with a patch date of December 5, 2018.  The data used in this article can be found here.

This research is an off-shoot of what I did with Android Auto, so if you want the full backstory, you can read the post here.  But…

Let’s Review

Google Assistant resides in the /data/data directory.  The folder is com.google.android.googlequicksearchbox.  See Figure 1.

galisting

Figure 1.

This folder also holds data about searches that are done from the Quick Search Box that resides at the top of my home screen (in Oreo).  This box has been lurking, in some fashion or another, since Doughnut, so it has had around 10 years or so to mature.  The folder has the usual suspect folders along with several others.  See Figure 2 for the folder listings.

galisting-infile

Figure 2.

The folder of interest here is app_session.  This folder has a great deal of data, but just looking at what is here one would not suspect anything.  The folder contains several .binarypb files, which I have learned, after having done additional research, are binary protocol buffer files.  These files are Google’s home-grown, XML-ish rival to JSON files.  They contain data that is relevant to how a user interacts with their device via Google Assistant.    See Figure 3.

binarypbs

Figure 3.

Each .binarypb file here represents a “session,” which I define as each time Google Assistant was invoked.  Based on my notes, I know when I summoned Google Assistant, how I summoned it, and what I did when I summoned it.  The first time I summoned Google Assistant is represented in the file with the last five digits of 43320.binarypb.  Figure 4 shows the header of the file.

car-readmessageheader

Figure 4.

The ASCII “car_assistant” seems to imply this request had been passed to Google Assistant from Android Auto.  In each test that I ran in Android Auto, this phrase appeared at the beginning of the file.  Additionally, the string in the smaller orange box (0x5951EF) appeared at the beginning of the file at the same byte offset each time (0x11).  I hesitate to call this a true “file header,” though.

If you read my Android Auto post, you will know the string in the red box is the start of a MP3 file.  You can see the end of the MP3 file in Figure 5.

lame

Figure 5.

The string in the orange box is the marker of the LAME MP3 codec, and the strings in the red boxes in Figures 4 and 5 are what I called “yoda” strings. Seeing these things, I carved from the first yoda (seen in Figure 4), to the last (seen in Figure 5), for a total of 11.1 KB.  I then saved the file with no extension and opened it in VLC Player.  The following came out of my speakers:

“You’ve got a few choices.  Pick the one you want.”

Based on my notes, this was the last phrase Google Assistant spoke to me via Android Auto prior to handing me off to Maps.  In this session, I had asked for directions to Starbucks and had not been specific about which one, which caused the returned reply that I had just heard.  There was other interesting data in this file, such as the text of what I had dictated to Google Assistant.  I began to wonder if it would be possible to determine if there were any patterns, identifying, or contextual data in this file that would be useful to or could act as “markers” for digital forensic practitioners.  Using 43320.binarypb as a starting point, I set off to see if I could map this file and the others on which I had taken notes.

Looking at these files in hex and ASCII, I started to notice a bit of a pattern.  While there is a difference between interactions in the car (via Android Auto) and outside of the car (yelling at the phone), there are some high-level similarities between these files regardless of how Google Assistant is used.  Below, I examine the data that is generated by Google Assistant via Android Auto.  The second part of this post will examine the data left behind when outside of the car.

A Deep Dive

I chose 43320.binarypb as my starting point on purpose:  there was a single request for directions in this session.  I thought the file would be straight forward, and I was right…sorta.

The session was started via Android Auto, and I had invoked Google Assistant via a button on my steering wheel (the phone was connected to the car).  The session went like this:

            Me:                  “I need directions to Starbucks.”

// Google Assistant thought for a few seconds //

            GA:                 “You’ve got a few choices.  Pick the one you want.”

After that I was handed off to Maps and presented with a few choices.  I chose a particular location and route, and then went on my way.  Figure 4 shows the top of the file, and I have already mentioned the MP3 data (Figures 4 and 5), so I will skip that portion of the file.

The first area of the file after the MP3 portion was a 4-byte string, BNDL (0x42444C02).  Just make note of this for now, because it comes up, a lot.  After BNDL there was some information about the version of Android Auto I was running, and, potentially, where the voice input was coming from (/mic /mic); see the red box and orange box, respectively, in Figure 6.

Figure 6

Figure 6.

There is an additional string in there that, if you weren’t paying attention, you would miss as it’s stuck at the end of some repetitive data.  I certainly missed it.  Take a look at the string in the blue box in Figure 6.  The string is 0x30CAF25768010000 (8 bytes) and appears at the end of some padding (please do not judge – I couldn’t come up with a better name for those 0xFF’s).  I read it little endian, converted it to decimal, and got a pleasant surprise:  1547663755824.  I recognized this format as Unix Epoch Time, so I turned to DCode, and had my Bob Ross-ian moment.  See Figure 7.

Side note:  I had been trying to find a date/time stamp in this file for two weeks, and, as frequently happens with me, I found it by accident. 

Figure 7

Figure 7.

Based on my notes, this is when I had summoned Google Assistant in order to ask for directions:  01/16/2019 at 13:35 (EST).

Next, com.google.android.apps.gsa.shared.search.QueryTriggerType (red box) caught my attention.  Just below it was the following:  webj gearhead* car_assistant gearhead (green box).   If you read my Android Auto post, you will know the title of the folder in which Android Auto resides has “gearhead” in it (com.google.android.projection.gearhead).  So, does this indicate Google Assistant was triggered via Android Auto?  Maybe…or maybe not.  This could be a one off.  I filed this away and continued.  See Figure 8.

Figure 8.PNG

Figure 8.

The next thing is something I mentioned in the Android Auto post.  A 5- byte string (0xBAF1C8F803) and an 8-byte string (0x014C604080040200) that appeared just above my actual vocal inquiry.  They can be seen in Figure 9:  the 5-byte string is in the blue box, the 8-byte string is in the green box, and the voice inquiry is in the top purple box.  Take note that there is a variation of what I actually said in the bottom purple box.  Also note the BNDL in the red box.

Figure 9

Figure 9.

Below that is data I had seen earlier in the file (in Figure 6):  the Android Auto version number, /mic /mic (orange box), the same time stamp I had just seen (purple box) QueryTriggerType with webj gearhead* car_assistant I need directions to Starbucks gearhead (green box).  And, there is BNDL again.  See Figures 10 and 11.

Figure 10

Figure 10.

Figure 11

Figure 11.

I want to draw attention to two additional things.  The first, in Figure 11, is another time stamp in the blue box.  This is another Unix time stamp (0xB176F15768010000).  This time is 01/16/2019 at 13:34:28 (EST). which is just under 1:30 earlier than the time stamp I had seen previously (when I invoked Google Assistant).  This is odd, but the string just below it may have something to do with it:  com.google.android.apps.gsa.shared.logger.latency.LatencyEvents.  I will say that 13:34 is when I connected the phone to the car and started Android Auto.

The second area is in the red box in Figure 12-1.  There you see the following:  velvet:query_state_search_result_id (red box) and then a 16-byte string ending in 0x12 (blue box).  This area appears in every Google Assistant session I have examined.  I have a theory about it but will wait until later to explain.  As with BNDL, just put it to the side for the moment.

Figure 12-1

Figure 12-1.

Figure 12-2

Figure 12-2.

In Figure 12 -1, you can also see BNDL (yellow box), the 8-byte green box string just prior to my vocal inquiry, and then the inquiry itself (purple box).  After a bit of padding, there is BNDL.   After that, in Figure 12-2, there is the same data seen in Figures 6 and 10 (orange box) , and…what’s this?  Another time stamp (red box)?  I did the same thing as before and got another Unix Epoch Time time stamp.

 As with the previous time stamp, this one is also prior to the first time stamp I had encountered in the file, although it is within the same minute in which I had invoked Google Assistant.  As before, this time stamp appears just before the string that contains LatencyEvents.  Does this have something to do with any latency the device is experiencing between it and Google’s servers?  Again, I am not sure.

Below this time stamp is a replay of what I had seen in Figure 10 (Figure 12-2 – orange box).  The area I discussed in Figure 11 is also present, sans my vocal input (purple).  See Figure 13.

Figure 13

Figure 13.

After that last BNDL, the same items I have already discussed are recycled again, and the first time stamp I had found is present again (red box).  See Figures 14-1, 14-2, and 14-3.

Figure 14-1.PNG

Figure 14-1.

Figure 14-2

Figure 14-2.

Figure 14-3

Figure 14-3.

The very last portion of the file is velvet:query_state:search_result_id (orange box) along with the 16-byte string (purple box); however, there is a small twist:  the last byte has changed from 0x12 to 0x18.  Just after that string is a 9-byte string, 0x01B29CF4AE04120A10 (blue box).  This string appears at the end of each session file I have examined, along with the string and.gsa.d.ssc (red box).  See Figure 15.

Figure 15

Figure 15.

So, just in this one file I saw patterns within the file, and recurring strings.  Were these things unique to this particular file, or does this pattern span across all of these files?

The next file I chose was 12067.binarypb.  As before, there was a single request for directions in this session.  This session, I was a bit more specific about the location for which I was looking.

This session was also started via Android Auto, and I had invoked Google Assistant via a button on my steering wheel (the phone was connected to the car).  The session went like this:

           Me:                  “Give me directions to the Starbucks in Fuquay Varina.”

                                    // Google Assistant thought for a few seconds //

           GA:                 “Starbucks is 10 minutes from your location by car and light traffic.”

As can be seen in Figure 16, the strings 0x5951EF and car_assistant can be seen at the top of the file.  Unlike the previous file, however, there is an additional bit of data here:  com.android.apps.gsa.search.core.al.a.au, a BNDL, and ICING_CONNECTION_INITIALIZATION_TIME_MSEC.  The “yoda” is also here.  See the blue, green, purple, orange, and red box, respectively, in Figure 16.

Figure 16

Figure 16.

Figures 17-1 and 17-2 show the end of the MP3 data, a BNDL, and then some data seen in the 43320.binarypb file:  the Android Auto version number, /mic /mic (orange box), a time stamp (red box), and QueryTriggerType with the webj gearhead* car_assistant gearhead (green box).  The time stamp here is 0x9FC2CB5B68010000, which, when converted to decimal, is 1547728306847.  Just like the previous file, this is also Unix Epoch Time.  I used DCode to convert, and got 01/17/2019 at 07:31:46 (EST).

Figure 17-1

Figure 17-1.

Figure 17-2

Figure 17-2.

According to my notes, this is the time I invoked Google Assistant, and asked for directions.

Traveling slightly further down I arrive in the area seen in Figure 18.  Here I find the 5-byte (blue box) and 8-byte strings (green box) I had seen in 43320.binarypb.  Then I see my request (purple box).  Also note the lower purple boxes; these appear to be what I said, and variations of what I said.  Just before each new variation, there is a number (4, 3, 5, and 9).  I will note that the text behind 4 and 5 differ only by the period at the end of the request.    I suspect that these numbers are assigned to each variation to keep tabs on each; however, I am not sure why.  There is also a BNDL at the end of this area (red box).

Figure 18

Figure 18.

Just below the requests I found some familiar information (Figures 19-1 and 19-2).  The Android Auto version number, /mic /mic (purple box), a time stamp (orange box), and QueryTriggerType with the webj gearhead* car_assistant gearhead (green box) are all here.  The time stamp here is the same as the previous one.  There is an additional piece of data here; just past the webj gearhead* car_assistant string is the 4give me directions to the Starbucks in Fuquay Varina gearhead (blue box).  There is also a BNDL at the end (red box).

Figure 19-1

Figure 19-1.

Figure 19-2

Figure 19-2.

Below the area in Figure 19-2, there is a time stamp (Figure 20) shown in a blue box.  The string (0xBC32C65B68010000) results in a Unix Epoch Time (1547727942332) of 01/17/2019 at 07:25:42 (EST), which is just under six minutes prior to my invocation of Google Assistant.  This time stamp appears just before com.google.android.apps.gsa.shared.logger.latency.LatencyEvents again.   This time coincided with me starting a podcast through Android Auto.

Figure 20

Figure 20.

Below the time stamp, the velvet:query_state_search_result_id appears again in Figures 21-1 and 21-2, along with the 16-byte string ending in 0x12 (green box) and a BNDL, the 8-byte string, and then my vocal inquiries and their variations, and another BNDL.  See the red, green, blue, purple, and orange boxes, respectively.

Figure 21-1

Figure 21-1.

Figure 21-2

Figure 21-2.

Just after the BNDL is the information about the Android Auto version I was using, the /mic /mic string (orange box), and a Unix Epoch Time time stamp (red box).  This one is the same as the first one I had seen in this file (the time I invoked Google Assistant).  See Figure 22.

Figure 22

Figure 22.

Below that are some new things.  First, the text of the MP3 file at the beginning of this file (purple box).  Second, a string that fits a pattern that I see in other files:  xxxxxxxxxxxx_xxxxxxxxx (green box).  The content of the string is different, but, most of the time, the format is 12 characters underscore 12 characters.  I am not sure what these are, so if any reader knows, please let me know so I can add it here (full credit given).  For the purposes of this article I will refer to it as an identifier string.

Also present is the URL for the location I asked for in Google Maps (orange box), and another identifier string (yellow box).  Beyond that, is the velvet:query_state_search_result_id string, along with the 16-byte string ending in 0x18 (red box), the 9-byte string (0x01B29CF4AE04120A10 – blue box), and the string and.gsa.d.ssc (yellow box).  See Figures 23-1 and 23-2.

Figure 23-1

Figure 23-1.

Figure 23-2

Figure 23-2.

So, for those keeping score, let’s review.  While each request was slightly different, there were some consistencies between both files.  The format, in particular, was fairly close:

  1. The beginning of the file (the 3-byte 0x5951EF string and “car_assistant).
  2. The MP3 audio at the beginning of the file which contains the last audio interaction prior to being sent to a different app (Maps).
  3. BNDL
  4. Android Auto Version along with the /mic /mic
  5. The date/time stamp of when Google Assistant is invoked, which appears just after some padding (0xFF).
  6. A 5-byte string (0xBAF1C8F803) that appears just before the vocal input appears the first time in a file. This string only appears here, and does not appear elsewhere.
  7. An 8-byte string (0x014C604080040200) that appears just before the vocal input, regardless of where it appears within the file.
  8. Text of the vocal input.
  9. BNDL
  10. Android Auto Version along with the /mic /mic
  11. Another date/time stamp of when Google Assistant was invoked (same as the first).
  12. The string webj gearhead* car_assistant <my vocal input> gearhead (what I actually said)
  13. BNDL
  14. What I have decided to call a “latency time stamp,” although, it may indicate the last time any activity was done via Android Auto (including starting Android Auto) prior to the invocation of Google Assistant.
  15. The velvet:query_state:search_result_id string appears along with the 16-byte string ending in 0x12.
  16. Items 7, 8, 9, 10, and 11 recycle.
  17. The velvet:query_state:search_result_id string appears along with the 16-byte string ending in 0x18, which appears at the end of the file.
  18. The 9-byte string 0x01B29CF4AE04120A10 after Item 17.
  19. The string gsa.d.ssc that appears after Item 18.

There is some divergence in the files.  In 43320, the items 7, 8, 9, 10, and 11 recycle a second time, whereas they recycle only once in 12067, and it also contains an extra latency time stamp that was not present in 12067.  Additionally, 12067 contains some extra data at the end of the file, specifically, the text of the MP3 file at the start of the file, an identifier string, a URL for Maps, and another identifier string.

File 12067 also had some extra data at the beginning that did not appear in 43320.

I also used Android Auto to test sending and receiving messages, and the file that represents that test is 22687.binarypb.  There were three sessions on 01/27/2019  The first session, which started at 14:16 (EST) went as follows:

Chat 1

About two minutes later at 14:16, a second session was started.  It went as follows:

Chat 2

About 3 minutes later (14:21 EST) I asked for directions using the same method as before (invocation via a button on my steering wheel).  The session went as follows:

Chat 3

The first thing I notice is there is a single binarypb file for 01/27/2019 (22687), even though there were three sessions.  Inspection of the file finds the 3-byte string, 0x5951EF, is present along with car_assistant string.  There is also a “yoda.”  See the orange, blue, and red boxes, respectively in Figure 24.  I carved from the yoda in Figure 24 to the end of the padding in Figure 25 (orange box).

FIgure 24

Figure 24.

Figure 25

Figure 25.

The following came out of my computer speakers:

“Smithfield Chicken and BBQ is 51 minutes from your location by car and light traffic.”

Now, this is interesting.  The first two sessions, which started at 14:16 and 14:18, did not include anything regarding directions.  The third session at 14:21 did involve directions.  I wonder if the fact that the three sessions were so close together that Google Assistant/Android just made one binarypb file to encompass all three sessions.  That would require more testing to confirm (or disprove) but is beyond the scope of this exercise and article.

Figures 26-1 and 26-2 shows the end of the MP3 data and some familiar data:  the Android Auto version information, /mic /mic, and a time stamp.  It also shows the QueryTriggerType and the webj gearhead* car_assistant gearhead string.  See the blue, orange, red, purple, respectively.  The time stamp here is 0x38CCC29068010000.  I converted to decimal (1548616911928), fired up DCode and got 01/27/2019 at 14:21:51 (EST).  This time is when I started the session in which I asked for directions.

Figure 26-1

Figure 26-1.

Figure 26-2

Figure 26-2.

Below that is some more familiar data.  The 5-byte string (0xBAF1C8F803) and 8-byte string (0x014C604080040200) appear (blue and green boxes in Figure 27-1), and there is the vocal input from my request for directions (that occurred roughly three minutes later).  There are also variations of the actual vocal input; each variation is designated by a letter (J, K, O, P, and M) (purple boxes).  After the variations, is the Android Auto version string, the /mic /mic string, and the same timestamp from before (orange and red boxes) (Figure 27-2).

Figure 27-1

Figure 27-1.

Figure 27-2

Figure 27-2.

The QueryTriggerType (red box) appears along with the webj gearhead* car_assistant J get me directions to the Smithfield Chicken & BBQ in Warsaw North Carolina gearhead (green box).  A BNDL appears (blue box), and then another time stamp (purple).  The byte string is 0x171DBC9068010000, and, in decimal is 1548616473879.  This converts to 01/27/2019 at 14:14 (EST), which is the time I hooked the phone to the car and started Android Auto.    See Figure 28.

Figure 28

Figure 28.

After that data is the velvet:query_state:search_result_id string, the accompanying 16-byte string ending in 0x12 (orange box), and a BNDL (blue box).  The 8-byte string (0x014C604080040200) appears (green box), my vocal input that started the first session (“read my newest text message” – purple box), and then a BNDL (blue box).  After that it is the Android Auto version, /mic /mic (yellow box), and a timestamp (red box).  See Figures 29-1 and 29-2.  The time stamp here is 0x0385BD9068010000, which is decimal 1548616566019.  When converted, it is 01/27/2019 at 14:16:06 (EST), which is the time of the first invocated session.

Figure 29-1

Figure 29-1.

Figure 29-2

Figure 29-2.

Also in Figure 29-2 is the QueryTriggerType and the webj gearhead* car_assistant (dark purple box) string.

Figure 30 has some new data in it.  A string appears that, while not completely similar, is something I had seen before.  It appears to be an identifier string:  GeQNOXLPoNc3n_QaG4J3QCw.  This is not 12 characters underscore 12 characters, but it is close.  Right after the identifier string is my vocal input “read my new text message.”  See the red box, and blue box, respectively in Figure 30.

Figure 30

Figure 30.

Figures 31-1 and 31-2 shows the two new text messages that were identified.  See the blue and red boxes.

Figure 31-1

Figure 31-1.

Figure 31-2

Figure 31-2.

Scrolling down a bit I find another identifier string:  eQNOXLPoNc3n_QaG4J3QCw (red box).  This identifier is the same as the first one, but without the leading “G.”  After this identifier is the velvet:query_state:search_result_id and the accompanying 16-byte string ending in 0x12 (orange box).  A BNDL appears at the end (green box).  See Figure 32.

Figure 32

Figure 32.

Next up is the 8-byte string (0x014C604080040200), and my next vocal input “read it.”  Just below my vocal input is the Android Auto version information, /mic /mic, and a time stamp.  Just below the time stamp is the QueryTriggerType and the webj gearhead* car_assistant gearhead strings (not pictured).  See the blue, orange, red, and purple boxes, respectively in Figures 33-1 and 33-2.  The time stamp here is 0xD796BD9068010000.  I converted it to decimal (1548616570583), fired up DCode and got 01/27/2019 at 14:16:10 (EST).  While I was not keeping exact time, this would seem to be when Google Assistant asked me whether or not I wanted to read the chat message from Josh Hickman.

Figure 33-1

Figure 33-1.

Figure 33-2

Figure 33-2.

There is another identifier string further down the file:  GeQNOXLPoNc3n_QaG4J3QCw and just below it my vocal input “read my newest text message.”  See the blue and red boxes, respectively, in Figure 34.  This is interesting.  Could it be that Google Assistant is associating this newest vocal input (“read it”) with the original request (“read my newest text message”) by way of the identifier string in order to know that the second request is related to the first?  Maybe.  This would definitely require some additional research if the case.

Figure 34

Figure 34.

Figures 35 and 36 show the text messages that were new when the request was made.

Figure 35

Figure 35.

Figure 36

Figure 36.

After some gobbly-goo, I found another identifier string:  gwNOXPm3FfKzggflxo7QDg (red box).  This format is completely different from the previous two I had seen.  Maybe this is an identifier for the vocal input “read it.”  Maybe it’s a transactional identifier…I am not sure.  See Figure 37.

Figure 37

Figure 37.

In Figure 37 you can also see the velvet:query_state:search_result_id and the 16-byte string ending in 0x12 (orange box), a BNDL, (blue box) the 8-byte string (green box), my next vocal input (purple box), and another BNDL.

Figure 38 shows familiar data:  Android Auto version, /mic /mic (green box), and a time stamp:  0x27BABD9068010000 (red box).  This converts to 1548616579623 in decimal, and 01/27/2019 at 14:16:19 in Unix Epoch Time.  As with the previous request, I wasn’t keeping exact time, but this would probably line up with when I said “Go on to the next one.”

Figure 38

Figure 38.

Figure 39 shows the QueryTriggerType string along with webj gearhead* car_assistant string.

Figure 39

Figure 39.

Figure 40 shows that identifier string again, and the vocal input that kicked off this session “read my newest text message.”  I am beginning to suspect this is actually some type of transactional identifier to associate “go on to the next one” with “read my newest text message.”

Figure 40

Figure 40.

Figures 41 and 42 show the new text messages.

Figure 41

Figure 41.

Figure 42

Figure 42.

There is another identifier string in Figure 43:  jwNOXMv_Ne_B_QbewpK4CQ (blue box).  This format is completely new compared to the previous ones.  Additionally, the velvet:query_state:search_result_id and the 16-byte string ending in 0x12 (orange box) appears, a BNDL (red box), along with the 8-byte string (green box) my next vocal input (purple box), and a BNDL.

Figure 43

Figure 43.

Figure 44 shows the Android Auto version, /mic /mic (blue box), and another time stamp (red box).  This time stamp is 0xC1ECBD9068010000, which converts to 1548616592577.  This is 01/27/2019 at 14:16:32 (EST).  This probably coincides with my vocal input “That’s it for now.” 

Figure 44

Figure 44.

Figure 45 has the QueryTriggerType and webj gearhead* car_assistant.

Figure 45

Figure 45.

Figure 46 shows a few things.  The first is the velvet:query_state:search_result_id and the 16-byte string ending in 0x12 (orange box).  The second thing is another identifier string, MgNOXNWTAtGp5wL03bHACg (blue box).  As before, this format does not match anything I have seen previously.

The third, and the most interesting part, is the start of the second session.  The only dividing line here is the velvet:query_state:search_result_id and the 16-byte string ending in 0x12, and BNDL (red box).  The green box is the 8-byte string, and the purple box contains my vocal input, “read my newest text messages.”  The purple boxes below are variations of what I said.

Figure 46

Figure 46.

Figure 47 shows the Android Auto version string, the /mic /mic string (blue box), and another time stamp (red box).  This time the stamp is 0x8C28C09068010000.  This converts to 1548616738956 in decimal, which is 01/27/2019 at 14:18:58 (EST) in Unix Epoch Time, which is the time I invoked Google Assistant for the second session.

Figure 47

Figure 47.

The next string that appears is the QueryTriggerType and webj gearhead* car_assistant strings.  See Figure 48.

Figure 48

Figure 48.

The next string is another identifier string.  This time, it is associated with my newest vocal input:  “read my newest text messages.” (blue box)  The string is GJgROXL2qNeSIggfk05CQCg (green box).  See Figure 49.

Figure 49

Figure 49.

Figures 50 and 51 show the text messages.

Figure 50

Figure 50.

Figure 51

Figure 51.

The next thing I see is a unique identifier string:  JgROXL2qNeSIggfk05CWCg (orange box).  This string is the same as the previous one (in Figure 49), but without the leading “G.”  This behavior is the same that I saw in the first session.  Beyond that there is the velvet:query_state:search_result_id and the 16-byte string ending in 0x12 (blue box), a BNDL (red box), the 8-byte string (green box), and my next vocal input, “hear it” (purple box), and another BNDL.  See Figure 52.

Figure 52

Figure 52.

Figure 53 shows the Android Auto version string, the /mic /mic string (blue box), and another time stamp (red box).  This time the stamp is 0x153AC09068010000.  This converts to 1548616743445 in decimal, which is 01/27/2019 at 14:19:03 (EST) in Unix Epoch Time, which would coincide with my vocal input “hear it.”

Figure 53

Figure 53.

The next string that appears is the QueryTriggerType and webj gearhead* car_assistant strings.  See Figure 54.

Figure 54

Figure 54.

Scrolling a bit finds an identifier string I have seen before:  GJgROXL2qNeSIggfk05CQCg (green box).  This is the first identifier seen in this session (the second one).  Just below it is the vocal input that started this session:  “read my newest text messages” (red box).  See Figure 55.

Figure 55

Figure 55.

Figures 56 and 57 show the messages that were new.

Figure 56

Figure 56.

Figure 57

Figure 57.

Figure 58 shows a pattern I have seen before.  First is another identifier string:  MAROXPPhAcvn_QaPpI24BA (orange box).  The second and third are velvet:query_state:search_result_id and the 16-byte string ending in 0x12 (blue box).  There is another BNDL (red box), the 8-byte string (green box), my next vocal input (purple box), “I’ll reply”, and another BNDL.  Also note the variations of what I said below my actual input (lower purple boxes).

Figure 58

Figure 58.

Figure 59 shows the Android Auto version string, the /mic /mic string (blue box), and another time stamp (red box).  This time the stamp is 0xD85CC09068010000.  This converts to 1548616752344 in decimal, which is 01/27/2019 at 14:19:12 (EST) in Unix Epoch Time, which would coincide with my vocal input “I’ll reply.”

Figure 59

Figure 59.

Figure 60 shows the next string that appears is the QueryTriggerType and webj gearhead* car_assistant strings.

Figure 60

Figure 60.

The next thing of interest is what is seen in Figure 61.  There is another identifier string, GPQROXMz3L6qOggfWpKeoCw (blue box).  This is not a string we have seen before.  Just below it is the vocal input that started this session (red box).

Figure 61

Figure 61.

I had to scroll quite a bit through some Klingon, but eventually I got to the area in Figure 62.  The red box shows another identifier string:  PQROXMz3L6qOggfWpKeoCw.  This is the same string that we saw in Figure 61, sans the leading “G.”  Again, this behavior is a pattern that we have seen in this particular file.  It causes my suspicion to grow that it is some type of transactional identifier that keeps vocal input grouped together.

Figure 62

Figure 62.

The second thing is velvet:query_state:search_result_id and the 16-byte string ending in 0x12 (blue box).  In Figure 63, there is another BNDL (red box), the 8-byte string (green box), and my next vocal input (the dictated message – purple box).  Also note the variations of what I said below my actual input (lower purple boxes).  Note that each variation is delineated by a character:  B, =, and >.

Figure 63

Figure 63.

Figure 64 shows the Android Auto version string, the /mic /mic string (blue box), and another time stamp (red box).  This time the stamp is 0x2192C09068010000.  This converts to 1548616765985 in decimal, which is 01/27/2019 at 14:19:25 (EST) in Unix Epoch Time, which would coincide with my dictation of a message to Google Assistant.

Figure 64

Figure 64.

Figure 65 shows the next string that appears is the QueryTriggerType and webj gearhead* car_assistant strings.

Figure 65

Figure 65.

The next thing of interest is what is seen in Figure 66.  There is an identifier string that we have seen before, GPQROXMz3L6qOggfWpKeoCw (blue box), and the initial vocal input that started this session (green box).  Again, I am beginning to think this is a method of keeping vocal inputs grouped within the same session.

Figure 66

Figure 66.

Scrolling through yet more Klingon, I find the area shown in Figure 67.  The blue box shows another identifier string:  RwROXPzPAcG5gge9s5n4DQ (red box).  This is a new identifier.  The velvet:query_state:search_result_id string and the 16-byte string ending in 0x12 (orange box) are also present.  There is another BNDL (blue box), the 8-byte string (green box), my next vocal input (purple box), “send it”, and another BNDL.

Figure 67

Figure 67.

Figure 68 shows the Android Auto version string, the /mic /mic string (blue box), and another time stamp (red box).  This time the stamp is 0x16BAC09068010000.  This converts to 1548616776214 in decimal, which is 01/27/2019 at 14:19:36 (EST) in Unix Epoch Time, which would coincide with my instructing Google Assistant to send the message I dictated.

Figure 68

Figure 68.

Figure 69 shows the next string that appears is the QueryTriggerType and webj gearhead* car_assistant strings.

Figure 69

Figure 69.

Figure 70 shows an identifier string, UQROXJ6aGKixggfh64qYDg (orange box), which is new.  Just below it is the velvet:query_state:search_result_id string and the 16-byte string ending in 0x12 (blue box) are also present.  There is another BNDL (red box), the 8-byte string (green box).  Below the 8-byte string is the vocal input that started the third session:  “get me directions to the Smithfield Chicken & BBQ in Warsaw North Carolina” (purple box).  Figure 71 shows the variations of my vocal input, which are identified by J, K, O, P, and M (purple boxes).  Below that is a BNDL.  This is the same data seen in Figure 27.

Figure 70

Figure 70.

Figure 71

Figure 71.

Figure 72 shows the Android Auto version string, the /mic /mic string (blue box), and another time stamp (red box).  This time the stamp is 0x38CCC29068010000.  This converts to 1548616911928 in decimal, which is 01/27/2019 at 14:21:51 (EST) in Unix Epoch Time.  This is the same time stamp seen in Figure 26.

Figure 72

Figure 72.

Figure 73 shows the next string that appears is the QueryTriggerType and webj gearhead* car_assistant strings.

Figure 73

Figure 73.

Scrolling down oh so slightly I find the text of the MP3 file at the beginning of the file (blue box), and the Google Maps URL for the location for which I had asked for directions (green box).  See Figure 74.  Figure 75 shows another identifier string, 1wROXOOFJc7j_Aa72aPAB (orange box).  After the identifier string is the velvet:query_state:search_result_id string.  Additionally, the string with the 16-byte string ending in 0x18 (green box), the 9-byte string (0x01B29CF4AE04120A10 – blue box), and the string and.gsa.d.ssc (red box) appear.

Figure 74

Figure 74.

Figure 75

Figure 75.

Comparisons

So, what is the final score?  If you’re still reading, let’s recap, and include what we found in the 22687.binarypb file.  The differences are in italics:

  1. The beginning of the file (the 3-byte 0x5951EF string and “car_assistant”).
  2. The MP3 audio at the beginning of the file which contains the last audio interaction prior to being sent to a different app (Maps).
  3. BNDL
  4. Android Auto Version along with the /mic /mic
  5. The date/time stamp of when Google Assistant is invoked, which appears just after some padding (0xFF). In the third file, 22687, the time stamp is the time for the third session.
  6. A 5-byte string (0xBAF1C8F803) that appears just before the vocal input appears the first time in a file. This string only appears here, and does not appear elsewhere. In the third file, 22687, this appeared before the first vocal input, which, as it turns out, is the vocal input that started the third session.
  7. An 8-byte string (0x014C604080040200) that appears just before the vocal input, regardless of where and how many times it appears within the file.
  8. Text of the vocal input.
  9. BNDL
  10. Android Auto Version along with the /mic /mic
  11. Another date/time stamp of when Google Assistant was invoked (same as the first).
  12. The string webj gearhead* car_assistant <my vocal input> gearhead (what I actually said). This item only appeared once in 22687 (the inquiry asking for directions).
  13. BNDL
  14. What I have decided to call a “latency time stamp,” although, it may indicate the last time any activity was done via Android Auto (including starting Android Auto) prior to the invocation of Google Assistant. In 22687, this only happened once.
  15. The velvet:query_state:search_result_id string appears along with the 16-byte string ending in 0x12.
  16. Items 7, 8, 9, 10, and 11 recycle.
  17. The velvet:query_state:search_result_id string appears along with the 16-byte string ending in 0x18, which appears at the end of the file.
  18. The 9-byte string 0x01B29CF4AE04120A10 after Item 17.
  19. The string gsa.d.ssc that appears after Item 18.

Conclusions

I feel comfortable enough at this point to draw a few conclusions based on my observations up to this point.

  1. Each binarypb file will start by telling you where the request is coming from (car_assistant).
  2. What is last chronologically is first in the binarypb file. Usually, this is Google Assistant’s response (MP3 file) to a vocal input just before being handed off to whatever service (e.g. Maps) you were trying to use.  The timestamp associated with this is also at the beginning of the file.
  3. A session can be broken down in to micro-sessions. I will call them vocal transactions.
  4. Vocal transactions have a visible line of demarcation by way of the 16-byte string ending in 0x12.
  5. A BNDL starts a vocal transaction, but also further divides the vocal transaction in to small chunks.
  6. The first vocal input in the binarypb file is marked by a 5-byte string: 0xBAF1C8F803, regardless of when, chronologically, it occurred in the session.
  7. Each vocal input is marked by an 8-byte string:   While the 5-byte string appears at the first in the binarypb file only, the 8-byte string appears just prior to each vocal input.
  8. When Google Assistant doesn’t think it understands you, it generates different variations of what you said…candidates…and then selects the one it thinks you said.  You can see thise in these files.
  9. In sessions where Google Assistant needs to keep things tidy, it will assign an identifier to vocal transactions. There does not appear to be any consistency (as far as I can tell) as to the format of these identifiers.
  10. The end of the final vocal transaction is marked by a 16-byte string ending in 0x18.

Figure 76 shows a visual version of the session files, over all, and Figure 77 shows the vocal transactions portion in more detail.

img_0075

Figure 76.

img_0074

Figure 77.

What’s Next?

So, there is definitely a format to these files, and I believe there are enough markers that someone could create a tool to parse them to make them easier to read and examine.  They contain what a device owner said…what they actually said…to their car/device.  This data could be extremely valuable to examiners and investigators, regardless of the venue in which they operate (civil or criminal).

Yes, this data could potentially be in a user’s Google Takeout data, and getting it that way would be slightly easier, although there would be a waiting period.  But, what if you do not have the authority to get said data?  This data could still be resident on the device and, potentially, available.

If I had scripting skills, I would try to see if something could be written to parse these files; however, alas, I do not.  I have minimal scripting skills.  Most of what I do is via Google and Cntrl/Cmd-C and Cntrl/Cmd-V.  If any reader can do this, and is willing, please let me know and I will post your product here and give full credit.  It would be awesome.

The second part of this post is forthcoming.  If you can’t wait, here’s a sneak peek:  there are some similarities…

Grab Your Glass of Milk! Android Oreo Image Now Available (8.x)

Now available for download is an Android Oreo (8.x) image, which was created using a LG Nexus 5X and a stock Android image. This image contains user populated data within the stock Android apps, along with user populated data within 25 non-stock apps.

As with the Android Nougat image, this one also includes robust documentation, a .ufd file, a list of hash values, and Takeout data from the ESPs who provided it.  This image is freely available to anyone who wants it for training, education, testing, or research.

Please note the image and related materials are hosted by Digital Corpora.  You can everthing here.

Android Nougat (7.x) Image

Last month I created a physical image of an Android 7.x test phone and made it available to the DFIR community for testing, education, and research.  As part of this project, robust documentation was created in order to document aspects of the test phone and all of the apps that were populated with user data along with said data.  Takeout data is also included for the ESPs that provided it.  A .ufd file is included for good measure.

Thanks to Devon Ackerman of AboutDFIR.com and Jessica Hyde of Magnet Forensics who both made it possible.

Please note that the image and related materials are hosted by Digital Corpora.  You can everything here.

Ka-Chow!!! Driving Android Auto

Recently I purchased a new car.  I am talking brand spankin’ new.  I had been looking for a compact SUV for a while because of a growing family, and I found it:  a 2019 Nissan Rogue.  I purchased it in 2018, so this would make the car, like, really new.  I was super excited as this was the first time I had ever purchased a new car.

While signing the novel-sized stack of paperwork that is part of any car purchase, the woman I was working with and I were chatting about all the bells and whistles that were in my newly purchased ride, and she mentioned that she had a term for newer cars:  laptops on wheels.  She was absolutely correct.  My car keeps track of all sorts of things:  gas mileage, proximity to objects, tire pressure, what I’m listening to on the radio, external temperature, and other things I probably don’t know about.    The on-board electronics are crazy compared to my first car (a 1985 Chevy S-10 pickup).  Additionally, my new car supports Apple’s CarPlay and Google’s Android Auto, the two automotive interfaces developed for the two major mobile software platforms

While I have been using CarPlay for some time now, I have never used Android Auto.  I was aware of its existence, but that was about it.  When I bought the car, I was in the middle of creating a clean Android image for the DFIR community, so I thought it would be great to have this in the image since phone-to-car interfaces will become more and more common.  Currently, there are 29 different auto manufacturers that have various models which support Android Auto, and the list continues to grow.  Additionally, there are after-market radio manufacturers (e.g. Pioneer, Kenwood, etc.) that are baking Android Auto in to their units, so I feel that this will become more common place as time goes on.

After using Android Auto for just over a week, I created my Android image, and immediately starting poking around to see what I could find.  Surprisingly, there is not much in the way of Android Auto itself; it projects certain Android apps on to your car’s in-dash receiver which allows you to interact with those apps.  My car has a mic (in the roof) that also worked with Android Auto and allows for voice commands/inquiries to Google Assistant without picking up the phone.  Now, Android Auto does have the ability to be used on-device, but, for the purposes here, I concentrated on the in-dash flavor.

The data for Android Auto itself is accessible via an Android backup or a file system extraction.  However, the good stuff, as is typical with Android, was sitting in the /data area, which, as we all know, is not easily accessible these days.  Thus, this disclosure:  the phone I was using was rooted using TWRP and Magisk.

The tests were conducted using a Nexus 5X running Android 8.1 with a patch level date of December 05, 2018.  The image file that contains the data discussed in this article can be downloaded here.

This post is a bit lengthy, but to get a full picture of what is going on behind the scenes and what artifacts may be available to an examiner, it is necessary

Setup

As I previously mentioned, this research involved the use of my new car.  The first thing to do was to download the app itself.  Android Auto does not come as a default application with the image I downloaded (bullhead OPM 7), and I am not sure if Android OEMs install it as part of their skinning process.  I checked with some co-workers who have Android devices, and Android Auto was not on their devices; those guys have Samsungs, and I do not know anyone who has anything else.

The setup process was slightly different than what I was used to with CarPlay.  With CarPlay it is a matter of plugging in my iPhone and driving off.  The phone and the car recognize each other, and I did not have to do anything beyond organizing the apps on the screen (like I do on my phone).  Android Auto is slightly different.  When I plugged the phone in to the car it immediately started Android Auto.  From there, it was like any typical Android app that is started for the first time:  the app permissions dance.  Android Auto requested the following:  Calendar, Contacts, Location, Microphone, Phone, and SMS.

Additionally, I had to pair it to my car via Bluetooth.  Again, this is not something that I had to do with CarPlay.  I took the defaults on everything, agreed to the terms of service, and proceeded onward.

External Inspection

The home screen is straight forward.  See Figure 1.

img_0364Figure 1.

The home screen displays various information via cards.  In Figure 1 you can see stuff that may be relevant to me.  Here you can see the weather and a frequent location.  If you scroll down, you can see new text messages that have not yet be read (pictured in Figure 2).

img_0365Figure 2.

The icons at the bottom of the screen correspond to the different applications that are available through Android Auto.  From left to right:  Maps (Figure 3), Telephone (Figure 4), the Android home button, various applications that can use Android Auto (Figures 5 and 6), and an exit mechanism to return to my radio’s native interface (Figure 7).

img_0369Figure 3.

img_0368Figure 4.

img_0366Figure 5.

img_0367Figure 6.

img_0372Figure 7.

Internal Examination

For just over a week, I drove around with Android Auto.  I got directions, had it read messages to me (that I sent myself – no one has the number assigned to the test device), and played podcasts.    After generating the data, I performed a physical extraction, and loaded it in to WinHex to see what I could find.

Navigating to the /data/data directory finds the Android Auto folder.  It is listed as com.google.android.projection.gearhead (Figure 8).

entry

Figure 8.

Inside the folder was a typical layout for an Android application (Figure 9).

infile

Figure 9.

The app_impl folder contained .apk files that appeared to be relevant to the operating of the application.  The cache folder acts as a “scratchpad” of sorts where the applications that can use Android Auto can deposit assets as needed.  In my case, there was album art from the podcasts that I listened to along with picture files that were used to display the weather conditions that were displayed.  These files had a .0 file extension.

The code_cache folder was empty.

The databases folder contained four files.  The first was CloudCards.db, which contained only one table.  The table contained one entry that had a URL that referred to a .png file of what was supposed to be a sun for the weather card that was displayed on the home screen.  This file had a corresponding journal file that was empty.

The second file in this folder was the google_analytics_v4 6308.db file.  This file has one table that contains any data:  properties.  The table contains the Google Analytics Client ID (explained below), the Tracking ID (tid – described as an ID that distinguishes to which Google Analytics property to send data), and information about the app version.  As with the CloudCards.db file, this file also has a corresponding journal file that is empty

The files folder contained two files.  The first is gaClientId, which is the Google Analytics Client ID.  The Client ID is a randomly generated UUID (version 4) that is used to anonymously identify a particular user or device.

There was a file named phenotype.lock, but the file had no contents.

The shared_prefs folder is where the interesting information resides.  See Figure 10 for a file listing.

shareprefsFigure 10.

The first file, app_state_shared_preferences.xml, contains the time Android Auto was last run stored in Unix Epoch Time with the tag “pref_lastrun.”  See Figure 11.

app_state_shared_preferencesFigure 11.

The next file of interest is the common_user_settings.xml.  This file contained the Bluetooth MAC address for my car.  The file here had one entry.  I did not have access to a second car that could run Android Auto, so I do not know if this file could contain multiple listings.  See Figure 12.

commonusersettingsFigure 12.

The last file in this folder is location_manager.xml.  See Figure 13.

location_managerFigure 13.

This file contains some interesting data.  For starters, it has the name of my car, “Nissan.”  The latitude and longitude geolocates to the location where I was when I last disconnected the test device from the car, and the time was the time I disconnected (in Unix Epoch Time).  As far as the value in the accuracy tag, I can only hypothesize; however, I can tell you the latitude and longitude are extremely close to where I was located when I disconnected the test device (I would estimate within three meters of where my car was located when I disconnected).

Other Data

My usage of Android Auto was limited to getting directions, reading messages, and playing podcasts, so I expected to find some references or artifacts to these activities.  My examination revealed none of that.  My next step was to examine the applications that actually did the work:  Google and Maps.

The main method by which I interacted with Android Auto was via voice commands, which is handled by Google Assistant.  For the purposes of this article I will not deep-dive in to Google Assistant; I have posted additional research in that area that utilized some of the data here.  However, I do want to highlight how Android Auto and Google interact in order to facilitate hands-free use while driving.

The Google app resides in the /data/data directory.  The folder is com.google.android.googlequicksearchbox.  See Figure 14.

galistingFigure 14.

The folder has the usual suspect folders along with several others.  See Figure 15 for the folder listings.

galisting-infileFigure 15.

The folder of interest here is app_session.  This folder has a great deal of data that is relevant to my usage of Android Auto.  It could also house data that is relevant to any number of ways which a user interacts with their device by the use of their voice.  The folder contains several .binarypb files.  See Figure 16.

binarypbsFigure 16.

Each .binarypb file here represents a “session,” which I will define as each time Google Assistant was invoked.  I will focus on the files that were generated by use of Android Auto; some of the files were generated as a result of a different manner of invocation for other purposes.  Based on my notes, I know when I summoned Google Assistant and what I did when I summoned it.  By comparing my notes to the MAC times associated with each .binarypb file I identified the applicable files. The first time I summoned Google Assistant is represented in the file with the last five digits of 43320.binarypb.  Figure 17 shows the beginning of the file.

firstsummonFigure 17.

The ASCII “car_assistant” seems to imply this request had been passed to Google Assistant from Android Auto.  In each test that I ran, this phrase appeared at the beginning of the file.  Additionally, the string in the smaller orange box (0x5951EF) accompanied the phrase at the same byte offset each time.  I hesitate to call this a true “file header,” though.  I think someone with more time in DFIR should make that call.

The data in the red boxes, though, caught my attention.

Here is a fun fact about me:  one of the jobs I held in college was that of a disc jockey.  During this time, digital audio workstations were really coming in to their own, so I sat in front of Pro Tools quite a bit; I saw a lot of audio files in hex/ASCII.  I continued to use Pro Tools extensively through the mp3 craze of the late 90’s and early 2000’s and had worked with mp3 files quite a bit.  While it had been some time, my eyeballs immediately saw the “yoda” string.  It also helps that I am a Star Wars fan.

The “yoda” string (0xFFF344C4) is a frame sync header for mp3 files.  I will not go in to detail, but just know this string tipped me off that something else, not expected, may be in this file.  I decided to scroll to the end of the gobbly-goo to see what I could fine.  Needless to say, I was surprised.  See Figure 18.

lameFigure 18.

The orange box is important.  In it is “LAME3.99.5.”  LAME is a free, open-source mp3 encoder (codec).  This particular version, 3.99.5, was released in February of 2012.  The additional yoda strings along with the 0x55 values in between is what happens when mp3 frames need to be padded (i.e. filled in order to round out a frame to fit the bit rate at which the audio is captured).  Each mp3 encoder handles padding differently, but the repetitive data in between yodas was indicative.  All of this further heightened my suspicion that what I was looking at was mp3 data.

So, I carved from the first yoda (seen in Figure 17), to the last (seen in Figure 18), for a total of 11.1 KB.  I then saved the file with no extension and opened it in VLC Player.  The following came out of the speakers:

“You’ve got a few choices.  Pick the one you want.”

Based on my notes, this was the last phrase Google Assistant spoke to me prior to handing me off to Maps.  In this session, I had asked for directions to Starbucks and had not been specific about which one, which caused the returned reply that I had just heard.

starbucks-2Figure 19.

Scrolling further in to the file I found the area shown in Figure 19.  The purple box with the arrow next to it was my actual voice inquiry via Android Auto, and the orange box contains the “car_assistant” string.  The strings in the blue and green boxes were interesting as I had seen them elsewhere while examining other data in this folder.  Also, there is a second phrase there (bottom purple box) that is a variation of what is in the first purple box.  Interesting…

The following morning, I headed to Starbucks again. This time I was a little more specific about which one I wanted.  I invoked Google Assistant, asked for directions, and navigated to the location.  This session is represented 12067.binarypb file.  See Figure 20.

starbucksreduxFigure 20.

The same “car_assistant” phrase was there, along with the 0x5951EF string. There is also another yoda.  Time to scroll…

starbucksredux-2Figure 21.

There is the LAME codec reference, the padding, and the final yoda in Figure 21.  As before, I carved, saved, and played in VLC Player.  I heard the following:

“Starbucks is ten minutes from your location by car and light traffic.”

Based on my notes, this was, again, the last thing I heard prior to Maps initializing and taking over.  I continued to scroll through the file to see if I could find the voice request I had originally given Google Assistant. See Figure 22.

starbucksredux-3Figure 22.

The strings in the blue and green boxes are here, but the string in the red box is different.  As before, the purple box with the arrow next to it is what I actually said, and, as with the previous request, there are multiple versions of what I said below.  It could be that the app/service is generating variations of what I said, assigning some score/weight to them, and then picking which one it thinks is what I said.   At this point, this is merely speculation on my part.  There is definitely more research that can be done in this area.

A couple of days later I had to pick up my child who had been on vacation with the grandparents the previous week.  As is normal, my parents and I meet halfway so it isn’t too burdensome on either party.  This time, I had a text message that needed to be read prior to starting the navigation.  This session is represented by the file 22686.binarypb.  See Figure 23.

car-readmessageheader Figure 23.

The usual suspects are here, so I repeated my steps with regards to carving out the mp3 file, and got the following audio:

“Smithfield Chicken and BBQ is 51 minutes from your location by car and light traffic.”

Now, this is interesting.  This audio file is, in fact, the last thing Google Assistant said to me prior to handing me off to Maps.  However, this audio file, as with the others, sits at the front of the .binarypb file.  This particular session started out by reading my text messages.  I decided to scroll through to see if I could find that part of the session.

readmynewtextmessages Figure 24.

The text in Figure 24 is what started the session.  The purple box with the arrow represents what I actually said when I initially invoked Google Assistant.  The second purple box, as before, is a variation of what I said.  Also, the string in the green box is present, but the string in the blue box is absent.  Figure 25 is the message that was read to me.  Notice that it added the contact information for the message; I sent it from my personal phone, which has an entry in the contacts app on the test device.

firstreadmessageFigure 25.

After the message was completed, Google Assistant asked if I wanted to reply.  You can see what I said in Figure 26.

iwillreplyFigure 26.

The string in the green box is present…again…and a variation of what I said is seen in the bottom purple box.  I then proceeded to dictate my message, which is seen in Figure 27.

dictatedmessageFigure 27.

You can see the string in the green box is present, and what I actually dictated is in the purple box with the arrow next to it.  The variations are in the purple boxes below the first.  The message was read back to me, and then I was asked if I wanted to send it.  See Figure 28.

senditFigure 28.

There’s that pesky green box again.  This time, apparently, Google Assistant felt that it understood what it was I said.  There are no variations of what I said here.

The next thing I asked Google Assistant for was directions.  See Figure 29.

bbq-1Figure 29.

Because of the length of my request, I could not fit everything into the screenshot, but I grabbed what I could.  The blue box has returned in this request, and the ever-present green box is here.  As usual, my actual request is marked by the purple arrow, with variations of what I said in the purple boxes below.  The result of this inquiry was the mp3 file I had carved earlier, and I was then handed off to Maps.

Just an observation here…the hex string in the green box showed up each time Google Assistant listened for input from me, and the hex string in the blue box showed up each time I asked for directions to a location.  Is the green box a header?  Maybe.  I’ll defer to someone who has more experience in this area.

The next place I looked was Maps.  Maps was a little thin in the way of unique data related to Android Auto.  The biggest thing I found were audio files that contained the set of navigation directions that were used when I asked for directions to Smithfield Chicken & BBQ.  Those files can be found in /data/data/com.google.android.apps.maps/apps_tts-temp folder.  These file names are numbers starting with 0 and are in chronological order.  There are no file extensions, but the file headers are “RIFF.”

Conclusions

In the way of artifacts, Android Auto leaves very little behind with regards to user data, which is not surprising.  The app merely functions as an interface…a gateway, if you will… to other apps and services in Android, and those apps keep the data to themselves; this makes sense as they are the ones doing all the work.   It does, however, reveal some basic data about its use and (at least) one vehicle to which it has been connected.

As vehicles get more and more complex, vehicle manufacturers will continue to add Android Auto as a stock option to their various models.  Additionally, Android Auto can be used on the device itself, without the need of an infotainment center.  While this ability was not tested specifically, it stands to reason that the artifacts left behind are similar.

The information found in the Google Assistant area of the phone is interesting and could be a great research project.  Hopefully, someone…ahem…will take it up.

Below you will find a chart that can be used a quick reference.  It contains information about the artifacts/data discussed in this article.

File Name

Location

Artifact

Relevance

google_analytics_v4 6308.db

com.google.android.projection.gearhead/files Google Analytics Client ID Google Analytics Client ID (gaClientID) – Randomly generated UUID that is used to anonymously identify a particular device or user.

app_state_shared_preference.xml

com.google.android.projection.gearhead/shared_prefs

pref_lastrun

Last time Android Auto was run

cache folder

com.google.android.projection.gearhead/

any

Applications that can use Android Auto will drop assets/resources in this folder

common_user_settings.xml

com.google.android.projection.gearhead/shared_prefs

Bluetooth MAC Address

Applications that can use Android Auto will drop assets/resources in this folder

common_user_settings.xml

com.google.android.projection.gearhead/shared_prefs

Car Name & Last Location

The name of the car paired, and the last location Android Auto was located

app_session folder

com.android.googlequicksearchbox/

Hands-free interaction

The binarypb files in this folder contain interaction with Google Assistant via Android Auto.  The file header will indicate when Google Assistant received a request via Android Auto (“car_assistant”)