Two Snaps and a Twist – An In-Depth (and Updated) Look at Snapchat on Android

 

There is an update to this post. It can be found after the ‘Conclusion’ section.

I was recently tasked with examining a two-year old Android-based phone which required an in-depth look at Snapchat. One of the things that I found most striking (and frustrating) during this examination was the lack of a modern, in-depth analysis of the Android version of the application beyond the tcspahn.db file, which, by the way, doesn’t exist anymore, and the /cache folder, which isn’t really used anymore (as far as I can tell). I found a bunch of things that discussed decoding encrypted media files, but this information was years old (Snapchat 5.x). I own the second edition of Learning Android Forensics by Skulkin, Tyndall, and Tamma, and while this book is great, I couldn’t find where they listed the version of Snapchat they examined or the version of Android they were using; what I found during my research for this post did not really match what was written in their book. A lot of things have changed.

Googling didn’t seem to help either; I just kept unearthing the older research. The closest I got was a great blog post by John Walther that examined Snapchat 10.4.0.54 on Android Marshmallow. Some of John’s post lined up with what I was seeing, while other parts did not.

WHAT’S THE BIG DEAL?

Snapchat averages 190 million users daily, which is just under half of the U.S. population, and those 190 million people send three billion snaps (pictures/videos) daily. Personally, I have the app installed on my phone, but it rarely sees any usage. Most of the time I use it on my kid, who likes the filters that alter his voice or requires that he stick out his tongue. He is particularly fond of the recent hot dog filter.

One of the appealing things about Snapchat is that direct messages (DMs) and snaps disappear after a they’re opened. While the app can certainly be used to send silly, ephemeral pictures or videos, some people find a way to twist the app for their own nefarious purposes.

There has been plenty written in the past about how some traces of activity are actually recoverable, but, again, nothing recent. I was surprised to find that there was actually more activity-related data left behind than I thought.

Before we get started just a few things to note (as usual). First, my test data was generated using a Pixel 3 running Android 9.0 (Pie) with a patch level of February 2019. Second, the version of Snapchat I tested is 10.57.0.0, which was the most current version as of 05/22/2019. Third, while the phone was not rooted, it did have TWRP, version 3.3.0-0, installed. Extracting the data was straight forward as I had the Android SDK Platform tools installed on my laptop. I booted into TWRP and then ran the following from the command line:

adb pull /data/data/com.snapchat.android

That’s it. The pull command dropped the entire folder in the same path as where the platform tools resided.

As part of this testing, I extracted the com.snapchat.android folder five different times over a period of 8 days as I wanted to see what stuck around versus what did not. I believe it is also important to understand the volatility of the data that is provided in this app. I think understanding the volatility will help investigators in the field and examiners understand exactly how much time, if any, they have before the data they are seeking is no longer available.

I will add that I tested two tools to see what they could extract: Axiom (version 3.0) and Cellebrite (UFED 4PC 7.18 and Physical Analyzer 7.19). Both tools failed to extract (parsing not included) any Snapchat data. I am not sure if this is a symptom of these tools (I hope not) or my phone. Regardless, both tools extracted nothing.

TWO SNAPS AND…SOME CHANGE

So, what’s changed? Quite a bit as far as I can tell. The storage location of where some of the data that we typically seek has changed. There are enough changes that I will not cover every single file/folder in Snapchat. I will just focus on those things that I think may be important for examiners and/or investigators.

One thing has not changed: the timestamp format. Unless otherwise noted, all timestamps discussed are in Unix Epoch.

The first thing I noticed is that the root level has some new additions (along with some familiar faces). The folders that appear to be new are “app_textures”, “lib”, and “no_backup.” See Figure 1.

Figure 1. Root level of the com.snapchat.android folder.

The first folder that may be of interest is one that has been of interest to forensicators and investigators since the beginning: “databases.” The first database of interest is “main.db.” This database replaces tcspahn.db as it now contains a majority of user data (again, tcspahn.db does not exist anymore). There is quite a bit in here, but I will highlight a few tables. The first table is “Feed.” See Figure 2.

Figure 2. The Feed.

This table contains the last action taken in the app. Specifically, the parties involved in that action (seen in Figure 2), what the action was, and when the action was taken (Figure 3). In Figure 4 you can even see which party did what. The column “lastReadTimestamp” is the absolute last action, and the column “lastReader” show who did that action. In this instance, I had sent a chat message from Fake Account 1 (“thisisdfir”) to Fake Account 2 (“hickdawg957”) and had taken a screenshot of the conversation using Fake Account 1. Fake Account 2 then opened the message.

Enter aFigure 3. Last action. caption

Figure 4. Who did what?
The second table is “Friend.” This table contains anyone who may be my friend. The table contains the other party’s username, user ID, display name, the date/time I added that person as a friend (column “addedTimestamp”), and the date/time the other person added me as a friend (column “reverseAddedTimestamp”). Also seen is any emojis that may be assigned to my friends. See Figures 5, 6, and 7.

Figure 5. Username, User ID, & Display Name.
Figure 6. Friendmojis (Emojis added to my Friends.

Figure 7. Timestamps for when I added friends and when they added me.

Note that the timestamps are for when I originally added the friend/the friend added me. The timestamps here translate back to dates in November of 2018, which is when I originally created the accounts during the creation of my Android Nougat image.

One additional note here. Since everyone is friends with the “Team Snapchat” account, the value for that entry in the “addedTimestamp” column is a good indicator of when the account you’re examining was created.

The next table is a biggie: Messages. I will say that I had some difficulty actually capturing data in this table. The first two attempts involved sending a few messages back and forth, letting the phone sit for a 10 or so minutes, and then extracting the data. In each of those instances, absolutely NO data was left behind in this table.

In order to actually capture the data, I had to leave the phone plugged in to the laptop, send some messages, screenshot the conversation quickly, and then boot into TWRP, which all happened in under two minutes time. If Snapchat is deleting the messages from this table that quickly, they will be extremely hard to capture in the future.

Figure 8 is a screenshot of my conversation (all occurred on 05/30/2019) taken with Fake Account 1 (on the test phone) and Figure 9 shows the table entries. The messages on 05/30/2019 start on Row 6.

Figure 8. A screenshot of the conversation.

Figure 9. Table entries of the conversation.

The columns “timestamp” and “seenTimestamp” are self-explanatory. The column “senderId” is the “id” column from the Friends table. Fake Account 1 (thisisdfir) is senderId 2 and Fake Account 2 (hickdawg957) is senderId 1. The column “feedRowId” tells you who the conversation participants are (beyond the sender). The values link back to the “id” column in the Feed table previously discussed. In this instance, the participants in the conversation are hickdawg957 and thisisdifr.

In case you missed it, Figure 8 actually has two saved messages between these two accounts from December of 2018. Information about those saved messages appear in Rows 1 and 2 in the table. Again, these are relics from previous activity and were not generated during this testing. This is an interesting find as I had completely wiped and reinstalled Android multiple times on this device since the those messages were sent, which leads me to speculate these messages may be saved server-side.

In Figure 10, the “type” column is seen. This column shows the type of message was transmitted. There are three “snap” entries here, but, based on the timestamps, these are not snaps that I sent or received during this testing.

Figure 10. The “types” of messages.
After the “type” column there is a lot of NULL values in a bunch of columns, but you eventually get to the message content, which is seen in Figure 11. Message content is stored as blob data. You’ll also notice there is a column “savedStates.” I am not sure exactly what the entries in the cells are referring to, but they line up with the saved messages.

Figure 11. Message (blob) content.

In Figure 12, I bring up one of the messages that I recently sent.

Figure 12. A sample message.

The next table is “Snaps.” This table is volatile, to say the least. The first data extraction I performed was on 05/22/2019 around 19:00. However, I took multiple pictures and sent multiple snaps on 05/21/2019 around lunch time and the following morning on 05/22/2019. Overall, I sent eight snaps (pictures only) during this time. Figure 13. Shows what I captured during my first data extraction.

Figure 13. I appear to be messing some snaps.
Of the eight snaps that I sent, only six appear in the table. The first two entries in the table pre-date when I started the testing (on 05/21/2019), so those entries are out (they came from Team Snapchat). The first timestamp is from the first snap I sent on 05/22/2019 at 08:24. The two snaps from 05/21/2019 are not here. So, within 24 hours, the data about those snaps had been purged.

On 05/25/2019 I conducted another data extraction after having received a snap and sending two snaps. Figure 14 shows the results.

Figure 14. A day’s worth of snaps.
The entries seen in Figure 13 (save the first two) are gone, but there are two entries there for the snaps I sent. However, there is no entry for the snap I received. I checked all of the tables and there was nothing. I received the snap at 15:18 that day, and performed the extraction at 15:51. Now, I don’t know for sure that a received snap would have been logged. I am sure, however, that it was not there. There may be more testing needed here.

Figure 15 shows the next table, “SendToLastSnapRecipients.” This table shows the user ID of the person I last sent a snap to in the “key” column, and the time at which I sent said snap.

Figure 15. The last snap recipient.

MEMORIES

During the entire testing period I took a total of 13 pictures. Of those 13, I saved 10 of them to “Memories.” Memories is Snapchat’s internal gallery, separate from the phone’s Photos app. After taking a picture and creating an overlay (if desired), you can choose to save the picture, which places it in Memories. If you were to decide to save the picture to your Photos app, Snapchat will allow you to export a copy of the picture (or video).

And here is a plus for examiners/investigators: items placed in Memories are stored server-side. I tested this by signing into Fake Account 1 from an iOS device, and guess what…all of the items I placed in Memories on the Pixel 3 appeared on the iOS device.

Memories can be accessed by swiping up from the bottom of the screen. Figure 16 shows the Snapchat screen after having taken a photo but before snapping (sending) it. Pressing the area in the blue box (bottom left) saves the photo (or video) to Memories. The area in the red box (upper right) are the overlay tools.

Figure 16. The Snapchat screen.

Figure 17 shows the pictures I have in my Memories. Notice that there are only 9 pictures (not 10). More on that in a moment.

Figure 17. My memories. It looks like I am short one picture.

The database memories.db stores relevant information about files that have been saved to Memories. The first table of interest is “memories_entry.” This table contains an “id,” the “snap_id,” and the date the snap was created. There are two columns regarding the time: “created_time” and “latest_created_time.” In Figure 18 there is a few seconds difference between the values in some cells in the two columns, but there are also a few that are the same value. In the cells where there are differences, the differences are negligible.

There is also a column titled “is_private” (seen in Figure 19). This column refers to the My Eyes Only (MEO) feature, which I will discuss shortly. For now, just know that the value of 1 indicates “yes.”

Figure 18. Memories entries.

Figure 19. My Eyes Only status.

(FOR) MY EYES ONLY

I have been seeing a lot of listserv inquires as of late regarding MEO. Cellebrite recently added support for MEO file recovery in Android as of Physical Analyzer 7.19 (iOS to follow), and, after digging around in the memories database, I can see why this would be an issue.

MEO allows a user to protect pictures or videos with a passcode; this passcode is separate from the user’s password for their Snapchat account. A user can opt to use a 4-digit passcode, or a custom alphanumeric passcode. Once a user indicates they want to place a media file in MEO, that file is moved out of the Memories area into MEO (it isn’t copied to MEO).

MEO is basically a private part of Memories. So, just like everything else in Memories, MEO items are also stored server-side. I confirmed this when I signed in to Fake Account 1 from the iOS device; the picture I saved to MEO on the Pixel 3 appeared in MEO on the iOS device. The passcode was the same, too. Snapchat says if a user forgets the passcode to MEO, they cannot help recover it. I’m not sure how true that is, but who knows.

If you recall, I placed 10 pictures in Memories, but Figure 17 only showed 9 pictures. That is because I moved one picture to MEO. Figure 20 shows my MEO gallery.

Figure 20. MEO gallery.

In the memories database, the table “memories_meo_confidential” contains entries about files that have been placed in MEO. See Figure 21.

Figure 21. MEO table in the memories database.

This table contains a “user_id,” the hashed passcode, a “master_key,” and the initialization vector (“iv”). The “master_key” and “initialization vector” are both stored in base64. And, the passcode….well, it has been hashed using bcrypt (ugh). I will add that Cellebrite reports Physical Analyzer 7.19 does have support for accessing MEO files, and, while I did have access to 7.19, I was not able to tell if it was able to access my MEO file since it failed to extract any Snapchat data.

The “user_id” is interesting: “dummy.” I have no idea what that is referring to, and I could not find it anywhere else in the data I extracted.

The next table is “memories_media.” This table. Does have a few tidbits of interesting data: another “id,” the size of the file (“size”), and what type of file (“format”). Since all of my Memories are pictures, all of the cells show “image_jpeg.” See Figures 22 and 23.

Figure 22. “memories_media.”

Figure 23. “memories_media,” part 2.

The next table is “memories_snap.” This table has a lot of information in about my pictures, and brings together data from the other tables in this database. Figure 24 shows a column “media_id,” which corresponds to the “id” in the “memories_media” table discussed earlier. There is also a “creation_time” and “time_zone_id” column. See Figure 24.

Figure 24. id, media_id, creation_time, and time zone.

Figure 25 shows the width and height of the pictures. Also note the column “duration.” The value is 3.0 for each picture. I would be willing to be that number could be higher or lower if the media were videos.

Figure 25 also shows the “memories_entry_id,” which corresponds to the “id” column in the “memories_entry” table. There is also a column for “has_location.” Each of the pictures I placed in Memories has location data associated with it (more on that in a moment).

Figure 25. Picture size, another id, and a location indicator.

Figure 26 is interesting as I have not been able to find the values in the “external_id” or “copy_from_snap_id” columns anywhere.

Figure 26. No clue here.

The data seen in Figure 27 could be very helpful in situations where an examiner/investigator thinks there may be multiple devices in play. The column “snap_create_user_agent” contains information on what version of Snapchat created the the snap, along with the Android version and, in my case, my phone model.

Figure 27. Very helpful.

The column “snap_capture_time” is the time I originally took the picture and not the time I sent the snap.

Figure 28 shows information about the thumbnail associated with each entry.

Figure 28. Thumbnail information.

Figure 29 is just like Figure 27 in its level of value. It contains latitude and longitude of the device when the picture was taken. I plotted each of these entries and I will say that the coordinates are accurate +/- 10 feet. I know the GPS capabilities of every device is different, so just be aware that your mileage may vary.

Figure 29. GPS coordinates!!

Figure 29 also has the column “overlay_size.” This is a good indication if a user has placed an overlay in the picture/video. Overlays are things that are placed in a photo/video after it has been captured. Figure 30 shows an example of an overlay (in the red box). The overlay here is caption text.

Figure 30. An overlay example.

If the value in the overlay_size column is NULL that is a good indication that no overlay was created.

Figure 31 shows the “media_key” and “media_iv,” both of which are in base64. Figure 32 shows the “encrypted_media_key” and “encrypted_media_iv” values. As you can see there is only one entry that has values for these columns; that entry is the picture I placed in MEO.

Figure 31. More base64.

Figure 32. Encrypted stuff.

The next table that may be of interest is “memories_remote_operation.” This shows all of the activity taken within Memories. In the “operation” column, you can see where I added the 10 pictures to Memories (ADD_SNAP_ENTRY_OPERATION). The 11th entry, “UPDATE_PRIVATE_ENTRY_OPERATION,” is where I moved a picture into MEO. See Figure 33.

Figure 33. Remote operations.

The column “serialized_operation” stores information about the operation that was performed. The data appears to be stored in JSON format. The cell contains a lot of the same data that was seen in the “memories_snap” table. I won’t expand it here, but DB Browser for SQLite does a good job of presenting it.

Figure 34 shows a better view of the column plus the “created_timestamp” column. This is the time of when the operation in the entry was performed.

Figure 34. JSON and a timestamp for the operation.

Figure 35 contains the “target_entry” column. The values in these columns refer to the “id”column in the “memories_entry” table.

Figure 35. Operation targets.

To understand the next database, journal, I first have to explain some additional file structure of the com.snapchat.android folder. If you recall all the way back to Figure 1, there was a folder labeled “files.” Entering that folder reveals the folders seen in Figure 36. Figure 37 shows the contents of the “file_manager” folder.

Figure 36. “Files” structure.

Figure 37. file_manager.

The first folder of interest here is “media_package_thumb,” the contents of which can be seen in Figure 38.

Figure 38. Thumbnails?

Examining the first file here in hex finds a familiar header: 0xFF D8 FF E0…yoya. These things are actually JPEGs. So, I opened a command line in the folder, typed ren *.* *.jpg and BAM: pictures! See Figure 39.

Figure 39. Pictures!

Notice there are a few duplications here. However, there are some pictures here that were not saved to memories and were not saved anywhere else. As an example, see the picture in Figure 40.

Figure 40. A non-saved, non-screenshot picture.
Figure 40 is a picture of the front of my employer’s building. For documentation purposes, I put a text overlay in the picture with the date/time I took it (to accompany my notes). I then snapped this picture to Fake Account 2, but did not save it to Memories, did not save it to my Photos app, and did not screenshot it. However, here it is, complete with the overlay. Now, while this isn’t the original picture (it is a thumbnail) it can still be very useful; one would need to examine the “snap” table in the main database to see if there was any activity around the MAC times for the thumbnail.

The next folder of interest is the “memories_media” folder. See Figure 41.

Figure 41. Hmm…

There are 10 items here. These are also JPEGs. I performed the same operation here as I did in the “media_package_thumb” folder and got the results seen in Figure 42.

Figure 42. My Memories, sans overlays.

These are the photographs I placed in Memories, but the caption overlays are missing. The picture that is MEO is also here (the file staring with F5FC6BB…). Additionally, these are high resolution pictures.

You may be asking yourself “What happened to the caption overlays?” I’m glad you asked. They are stored in the “memories_overlay” folder. See Figure 43.

Figure 43. My caption overlays.

Just like the previous two folders, these are actually JPEGs. I performed the rename function, and got the results seen in Figure 44. Figure 45 shows the overlay previously seen in Figure 30.

Figure 44. Overlays.

Figure 45. The Megaman overlay from Figure 30.

The folder “memories_thumbnail” is the same as the others, except it contains just the files in Memories (with the overlays). For brevity’s sake, I will just say the methodology to get the pictures to render is the same as before. Just be aware that while I just have pictures in my Memories, a user could put videos in there, too, so you could have a mixture of media. If you do a mass-renaming, and a file does not render, the file extension is probably wrong, so adjust the file extension(s) accordingly.

Now that we have discussed those file folders, let’s get back to the journal database. This database keeps track of everything in the “file_manager” directory, including those things we just discussed. Figure 46 shows the top level of the database’s entries.

Figure 46. First entries in the journal database.

If I filter the “key” column using the term “package” from the “media_package_thumb” folder (the “media_package_thumb.0” files) I get the results seen in Figure 47.

Figure 47. Filtered results.

The values in the “key” column are the file names for the 21 files seen in Figure 38. The values seen in the “last_update_time” column are the timestamps for when I took the pictures. This is a method by which examiners/investigators could potentially recover snaps that have been deleted.

WHAT ELSE IS THERE?

As it turns out, there are a few more, non-database artifacts left behind which are located in the “shared_prefs” folder seen in Figure 1. The contents can be seen in Figure 48.

Figure 48. shared_prefs contents.

The first file is identity_persistent_store.xml seen in Figure 49. The file contains the timestamp for when Snapchat was installed on the device (INSTALL_ON_DEVICE_TIMESTAMP), when the first logon occurred on the device (FIRST_LOGGED_IN_ON_DEVICE_TIMESTAMP), and the last user to logon to the device (LAST_LOGGED_IN_USERNAME).

Figure 49. identity_persistent_store.xml.

Figure 50. shows the file LoginSignupStore.xml. it contains the username that is logged in.

Figure 50. Who is logged in?

The file user_session_shared_pref.xml has quite a bit of account data in it, and is seen in Figure 51. For starters, it contains the display name (key_display_name), the username (key_username), and the phone number associated with the account (key_phone).

The value “key_created_timestamp” is notable. This time stamp converts to November 29, 2018 at 15:13:34 (EST). Based on my notes from my Nougat image, this was around the time I established Fake Account 1, which was used in the creation of the Nougat image. This might be a good indicator of when the account was established, although, you could always get that data from serving Snapchat with legal process.

Rounding it out is the “key_user_id” (seen in the Friends table of the main database) and the email associated with the account (key_email).

Figure 51. user_session_shared_pref.xml

CONCLUSION

Snapchat’s reputation proceeds it very well. I have been in a few situations where examiners/investigators automatically threw up their hands and gave up after having been told that potential evidence was generated/contained in Snapchat. They wouldn’t even try. I will say that while I always have (and will) try to examine anything regardless of what the general concensus is, I did share a bit of others’ skepticism about the ability to recover much data from Snapchat. However, this exercise has shown me that there is plenty of useful data left behind by Snapchat that can give a good look into its usage.

Update

Alexis Brignoni over at Initialization Vectors noticed that I failed to address something in this post. First, thanks to him for reading and contacting me. 🙂 Second, he noticed that I did not address Cellebrite Physical Analyzer’s (v 7.19) and Axiom’s (v 3.0) ability to parse my test Snapchat data (I addressed the extraction portion only).

We both ran the test data against both tools and found both failed to parse any of the databases. Testing found that while Cellebrite found the pictures I describe in this post, it did not apply the correct MAC times to them (from the journal.db). Axiom failed to parse the databases and failed to identify any of the pictures.

This is not in any way shape or form a knock on or an attempt to single out these two tools; these are just the tools to which I happen to have access. These tools work, and I use them regularly. The vendors do a great job keeping up with the latest developments in both the apps and the operating systems. Sometimes, though, app developers will make a hard turn all of a sudden, and it does take time for the vendors to update their tools. Doing so requires R&D and quality control via testing, which can take a while depending on the complexity of the update.

However, this exercise does bring to light an important lesson in our discipline, one that bears repeating: test and know the limitations of your tools. Knowing the limitations allows you to know when you may be missing data/getting errant readings. Being able to compensate for any shortcomings and manually examine the data is a necessary skillset in our discipline.

Thank you Alexis for the catch and assist!

OK Computer…er…Google. Dissecting Google Assistant (Part 1)

A few weeks ago, I posted a blog about some research I conducted on Android Auto, and I mentioned there was some interesting data left behind by Google Assistant when using Android Auto.  Based on what I found there, I decided to go further down the virtual assistant rabbit hole to see what I could find.

As far as virtual assistants go, I use Siri.  When I had a newborn, Siri was used a lot.  In addition to turning on/off lights or playing music, I used Siri to turn on certain appliances (via smart plugs), respond to texts, and make phone calls as my hands were usually holding a baby, trying to do something for/to/with a baby, or I was trying to help my spouse do those things.  Siri was really helpful then.  Siri is still useful, but, nowadays, my primary use of Siri is in the car.  There are times where I still yell at my phone or HomePod to find a TV show, play a song, turn on a light, or answer a quick math question.  For other things such as search and typing a message (outside of the car), I’m old fashioned.

I have been fascinated by talking computers/A.I. for a long time.  According to my parents, I used to love Knight Rider.  It had the Knight Industries Two Thousand (K.I.T.T.) – the snarky, crime-fighting automotive sidekick of the Hoff.  As you can see from the GIF above, I also like Star Trek.  Captain Kirk, Spock, et al. have been verbally interacting with their computers since the 1960’s.  WOPR/Joshua, check.  Project 2501, check.  The Architect, check.  And, the crème de la crème:  HAL 9000, check check check.

While researching Google Assistant, I stumbled across an article that had some interesting statistics.  In 2017, there was a 128.9% year-over -year increase in the use of voice-activated assistants, with an expected growth of 40% in 2019.  Another statistic:  2 out of 5 adults use voice search at least once a day; this was in 2017, so I suspect this number is higher by now.  This explosion, in my opinion, started when Alexa arrived.  Say what you like about Amazon, but they were smart to open Alexa to developers.  Alexa is everywhere and owns around 70% of the virtual assistant market.  With Amazon’s recent acquisition of eero (to my dismay), wireless routers, the televisions of the 21st century, will have Alexa in them.  Alexa. Is. Everywhere.  I am sure Google will follow, so I feel it is important to understand any artifact Google Assistant may leave behind on a device.

There are three things to be aware of before I get started.  First, the data I found resides in the /data/ directory, and, thus, is not easily accessible on newer devices unless the device is rooted, or you have a tool that can access this area.  Second, a warning:  this post is lengthy, but in order to understand the patterns and data in the three files I examine, it is necessary.

Finally, this will be a two-parter.  There was way too much data to cover everything in just one post.  This post will examine the data left behind by Google Assistant when used via Android Auto.  The second post will examine the data left behind by Google Assistant when used outside of the car (i.e. when I yell at the device).

One final note:  this data was generated on a rooted Nexus 5X running Android Oreo (8.1), with a patch date of December 5, 2018.  The data used in this article can be found here.

This research is an off-shoot of what I did with Android Auto, so if you want the full backstory, you can read the post here.  But…

Let’s Review

Google Assistant resides in the /data/data directory.  The folder is com.google.android.googlequicksearchbox.  See Figure 1.

galisting

Figure 1.

This folder also holds data about searches that are done from the Quick Search Box that resides at the top of my home screen (in Oreo).  This box has been lurking, in some fashion or another, since Doughnut, so it has had around 10 years or so to mature.  The folder has the usual suspect folders along with several others.  See Figure 2 for the folder listings.

galisting-infile

Figure 2.

The folder of interest here is app_session.  This folder has a great deal of data, but just looking at what is here one would not suspect anything.  The folder contains several .binarypb files, which I have learned, after having done additional research, are binary protocol buffer files.  These files are Google’s home-grown, XML-ish rival to JSON files.  They contain data that is relevant to how a user interacts with their device via Google Assistant.    See Figure 3.

binarypbs

Figure 3.

Each .binarypb file here represents a “session,” which I define as each time Google Assistant was invoked.  Based on my notes, I know when I summoned Google Assistant, how I summoned it, and what I did when I summoned it.  The first time I summoned Google Assistant is represented in the file with the last five digits of 43320.binarypb.  Figure 4 shows the header of the file.

car-readmessageheader

Figure 4.

The ASCII “car_assistant” seems to imply this request had been passed to Google Assistant from Android Auto.  In each test that I ran in Android Auto, this phrase appeared at the beginning of the file.  Additionally, the string in the smaller orange box (0x5951EF) appeared at the beginning of the file at the same byte offset each time (0x11).  I hesitate to call this a true “file header,” though.

If you read my Android Auto post, you will know the string in the red box is the start of a MP3 file.  You can see the end of the MP3 file in Figure 5.

lame

Figure 5.

The string in the orange box is the marker of the LAME MP3 codec, and the strings in the red boxes in Figures 4 and 5 are what I called “yoda” strings. Seeing these things, I carved from the first yoda (seen in Figure 4), to the last (seen in Figure 5), for a total of 11.1 KB.  I then saved the file with no extension and opened it in VLC Player.  The following came out of my speakers:

“You’ve got a few choices.  Pick the one you want.”

Based on my notes, this was the last phrase Google Assistant spoke to me via Android Auto prior to handing me off to Maps.  In this session, I had asked for directions to Starbucks and had not been specific about which one, which caused the returned reply that I had just heard.  There was other interesting data in this file, such as the text of what I had dictated to Google Assistant.  I began to wonder if it would be possible to determine if there were any patterns, identifying, or contextual data in this file that would be useful to or could act as “markers” for digital forensic practitioners.  Using 43320.binarypb as a starting point, I set off to see if I could map this file and the others on which I had taken notes.

Looking at these files in hex and ASCII, I started to notice a bit of a pattern.  While there is a difference between interactions in the car (via Android Auto) and outside of the car (yelling at the phone), there are some high-level similarities between these files regardless of how Google Assistant is used.  Below, I examine the data that is generated by Google Assistant via Android Auto.  The second part of this post will examine the data left behind when outside of the car.

A Deep Dive

I chose 43320.binarypb as my starting point on purpose:  there was a single request for directions in this session.  I thought the file would be straight forward, and I was right…sorta.

The session was started via Android Auto, and I had invoked Google Assistant via a button on my steering wheel (the phone was connected to the car).  The session went like this:

            Me:                  “I need directions to Starbucks.”

// Google Assistant thought for a few seconds //

            GA:                 “You’ve got a few choices.  Pick the one you want.”

After that I was handed off to Maps and presented with a few choices.  I chose a particular location and route, and then went on my way.  Figure 4 shows the top of the file, and I have already mentioned the MP3 data (Figures 4 and 5), so I will skip that portion of the file.

The first area of the file after the MP3 portion was a 4-byte string, BNDL (0x42444C02).  Just make note of this for now, because it comes up, a lot.  After BNDL there was some information about the version of Android Auto I was running, and, potentially, where the voice input was coming from (/mic /mic); see the red box and orange box, respectively, in Figure 6.

Figure 6

Figure 6.

There is an additional string in there that, if you weren’t paying attention, you would miss as it’s stuck at the end of some repetitive data.  I certainly missed it.  Take a look at the string in the blue box in Figure 6.  The string is 0x30CAF25768010000 (8 bytes) and appears at the end of some padding (please do not judge – I couldn’t come up with a better name for those 0xFF’s).  I read it little endian, converted it to decimal, and got a pleasant surprise:  1547663755824.  I recognized this format as Unix Epoch Time, so I turned to DCode, and had my Bob Ross-ian moment.  See Figure 7.

Side note:  I had been trying to find a date/time stamp in this file for two weeks, and, as frequently happens with me, I found it by accident. 

Figure 7

Figure 7.

Based on my notes, this is when I had summoned Google Assistant in order to ask for directions:  01/16/2019 at 13:35 (EST).

Next, com.google.android.apps.gsa.shared.search.QueryTriggerType (red box) caught my attention.  Just below it was the following:  webj gearhead* car_assistant gearhead (green box).   If you read my Android Auto post, you will know the title of the folder in which Android Auto resides has “gearhead” in it (com.google.android.projection.gearhead).  So, does this indicate Google Assistant was triggered via Android Auto?  Maybe…or maybe not.  This could be a one off.  I filed this away and continued.  See Figure 8.

Figure 8.PNG

Figure 8.

The next thing is something I mentioned in the Android Auto post.  A 5- byte string (0xBAF1C8F803) and an 8-byte string (0x014C604080040200) that appeared just above my actual vocal inquiry.  They can be seen in Figure 9:  the 5-byte string is in the blue box, the 8-byte string is in the green box, and the voice inquiry is in the top purple box.  Take note that there is a variation of what I actually said in the bottom purple box.  Also note the BNDL in the red box.

Figure 9

Figure 9.

Below that is data I had seen earlier in the file (in Figure 6):  the Android Auto version number, /mic /mic (orange box), the same time stamp I had just seen (purple box) QueryTriggerType with webj gearhead* car_assistant I need directions to Starbucks gearhead (green box).  And, there is BNDL again.  See Figures 10 and 11.

Figure 10

Figure 10.

Figure 11

Figure 11.

I want to draw attention to two additional things.  The first, in Figure 11, is another time stamp in the blue box.  This is another Unix time stamp (0xB176F15768010000).  This time is 01/16/2019 at 13:34:28 (EST). which is just under 1:30 earlier than the time stamp I had seen previously (when I invoked Google Assistant).  This is odd, but the string just below it may have something to do with it:  com.google.android.apps.gsa.shared.logger.latency.LatencyEvents.  I will say that 13:34 is when I connected the phone to the car and started Android Auto.

The second area is in the red box in Figure 12-1.  There you see the following:  velvet:query_state_search_result_id (red box) and then a 16-byte string ending in 0x12 (blue box).  This area appears in every Google Assistant session I have examined.  I have a theory about it but will wait until later to explain.  As with BNDL, just put it to the side for the moment.

Figure 12-1

Figure 12-1.

Figure 12-2

Figure 12-2.

In Figure 12 -1, you can also see BNDL (yellow box), the 8-byte green box string just prior to my vocal inquiry, and then the inquiry itself (purple box).  After a bit of padding, there is BNDL.   After that, in Figure 12-2, there is the same data seen in Figures 6 and 10 (orange box) , and…what’s this?  Another time stamp (red box)?  I did the same thing as before and got another Unix Epoch Time time stamp.

 As with the previous time stamp, this one is also prior to the first time stamp I had encountered in the file, although it is within the same minute in which I had invoked Google Assistant.  As before, this time stamp appears just before the string that contains LatencyEvents.  Does this have something to do with any latency the device is experiencing between it and Google’s servers?  Again, I am not sure.

Below this time stamp is a replay of what I had seen in Figure 10 (Figure 12-2 – orange box).  The area I discussed in Figure 11 is also present, sans my vocal input (purple).  See Figure 13.

Figure 13

Figure 13.

After that last BNDL, the same items I have already discussed are recycled again, and the first time stamp I had found is present again (red box).  See Figures 14-1, 14-2, and 14-3.

Figure 14-1.PNG

Figure 14-1.

Figure 14-2

Figure 14-2.

Figure 14-3

Figure 14-3.

The very last portion of the file is velvet:query_state:search_result_id (orange box) along with the 16-byte string (purple box); however, there is a small twist:  the last byte has changed from 0x12 to 0x18.  Just after that string is a 9-byte string, 0x01B29CF4AE04120A10 (blue box).  This string appears at the end of each session file I have examined, along with the string and.gsa.d.ssc (red box).  See Figure 15.

Figure 15

Figure 15.

So, just in this one file I saw patterns within the file, and recurring strings.  Were these things unique to this particular file, or does this pattern span across all of these files?

The next file I chose was 12067.binarypb.  As before, there was a single request for directions in this session.  This session, I was a bit more specific about the location for which I was looking.

This session was also started via Android Auto, and I had invoked Google Assistant via a button on my steering wheel (the phone was connected to the car).  The session went like this:

           Me:                  “Give me directions to the Starbucks in Fuquay Varina.”

                                    // Google Assistant thought for a few seconds //

           GA:                 “Starbucks is 10 minutes from your location by car and light traffic.”

As can be seen in Figure 16, the strings 0x5951EF and car_assistant can be seen at the top of the file.  Unlike the previous file, however, there is an additional bit of data here:  com.android.apps.gsa.search.core.al.a.au, a BNDL, and ICING_CONNECTION_INITIALIZATION_TIME_MSEC.  The “yoda” is also here.  See the blue, green, purple, orange, and red box, respectively, in Figure 16.

Figure 16

Figure 16.

Figures 17-1 and 17-2 show the end of the MP3 data, a BNDL, and then some data seen in the 43320.binarypb file:  the Android Auto version number, /mic /mic (orange box), a time stamp (red box), and QueryTriggerType with the webj gearhead* car_assistant gearhead (green box).  The time stamp here is 0x9FC2CB5B68010000, which, when converted to decimal, is 1547728306847.  Just like the previous file, this is also Unix Epoch Time.  I used DCode to convert, and got 01/17/2019 at 07:31:46 (EST).

Figure 17-1

Figure 17-1.

Figure 17-2

Figure 17-2.

According to my notes, this is the time I invoked Google Assistant, and asked for directions.

Traveling slightly further down I arrive in the area seen in Figure 18.  Here I find the 5-byte (blue box) and 8-byte strings (green box) I had seen in 43320.binarypb.  Then I see my request (purple box).  Also note the lower purple boxes; these appear to be what I said, and variations of what I said.  Just before each new variation, there is a number (4, 3, 5, and 9).  I will note that the text behind 4 and 5 differ only by the period at the end of the request.    I suspect that these numbers are assigned to each variation to keep tabs on each; however, I am not sure why.  There is also a BNDL at the end of this area (red box).

Figure 18

Figure 18.

Just below the requests I found some familiar information (Figures 19-1 and 19-2).  The Android Auto version number, /mic /mic (purple box), a time stamp (orange box), and QueryTriggerType with the webj gearhead* car_assistant gearhead (green box) are all here.  The time stamp here is the same as the previous one.  There is an additional piece of data here; just past the webj gearhead* car_assistant string is the 4give me directions to the Starbucks in Fuquay Varina gearhead (blue box).  There is also a BNDL at the end (red box).

Figure 19-1

Figure 19-1.

Figure 19-2

Figure 19-2.

Below the area in Figure 19-2, there is a time stamp (Figure 20) shown in a blue box.  The string (0xBC32C65B68010000) results in a Unix Epoch Time (1547727942332) of 01/17/2019 at 07:25:42 (EST), which is just under six minutes prior to my invocation of Google Assistant.  This time stamp appears just before com.google.android.apps.gsa.shared.logger.latency.LatencyEvents again.   This time coincided with me starting a podcast through Android Auto.

Figure 20

Figure 20.

Below the time stamp, the velvet:query_state_search_result_id appears again in Figures 21-1 and 21-2, along with the 16-byte string ending in 0x12 (green box) and a BNDL, the 8-byte string, and then my vocal inquiries and their variations, and another BNDL.  See the red, green, blue, purple, and orange boxes, respectively.

Figure 21-1

Figure 21-1.

Figure 21-2

Figure 21-2.

Just after the BNDL is the information about the Android Auto version I was using, the /mic /mic string (orange box), and a Unix Epoch Time time stamp (red box).  This one is the same as the first one I had seen in this file (the time I invoked Google Assistant).  See Figure 22.

Figure 22

Figure 22.

Below that are some new things.  First, the text of the MP3 file at the beginning of this file (purple box).  Second, a string that fits a pattern that I see in other files:  xxxxxxxxxxxx_xxxxxxxxx (green box).  The content of the string is different, but, most of the time, the format is 12 characters underscore 12 characters.  I am not sure what these are, so if any reader knows, please let me know so I can add it here (full credit given).  For the purposes of this article I will refer to it as an identifier string.

Also present is the URL for the location I asked for in Google Maps (orange box), and another identifier string (yellow box).  Beyond that, is the velvet:query_state_search_result_id string, along with the 16-byte string ending in 0x18 (red box), the 9-byte string (0x01B29CF4AE04120A10 – blue box), and the string and.gsa.d.ssc (yellow box).  See Figures 23-1 and 23-2.

Figure 23-1

Figure 23-1.

Figure 23-2

Figure 23-2.

So, for those keeping score, let’s review.  While each request was slightly different, there were some consistencies between both files.  The format, in particular, was fairly close:

  1. The beginning of the file (the 3-byte 0x5951EF string and “car_assistant).
  2. The MP3 audio at the beginning of the file which contains the last audio interaction prior to being sent to a different app (Maps).
  3. BNDL
  4. Android Auto Version along with the /mic /mic
  5. The date/time stamp of when Google Assistant is invoked, which appears just after some padding (0xFF).
  6. A 5-byte string (0xBAF1C8F803) that appears just before the vocal input appears the first time in a file. This string only appears here, and does not appear elsewhere.
  7. An 8-byte string (0x014C604080040200) that appears just before the vocal input, regardless of where it appears within the file.
  8. Text of the vocal input.
  9. BNDL
  10. Android Auto Version along with the /mic /mic
  11. Another date/time stamp of when Google Assistant was invoked (same as the first).
  12. The string webj gearhead* car_assistant <my vocal input> gearhead (what I actually said)
  13. BNDL
  14. What I have decided to call a “latency time stamp,” although, it may indicate the last time any activity was done via Android Auto (including starting Android Auto) prior to the invocation of Google Assistant.
  15. The velvet:query_state:search_result_id string appears along with the 16-byte string ending in 0x12.
  16. Items 7, 8, 9, 10, and 11 recycle.
  17. The velvet:query_state:search_result_id string appears along with the 16-byte string ending in 0x18, which appears at the end of the file.
  18. The 9-byte string 0x01B29CF4AE04120A10 after Item 17.
  19. The string gsa.d.ssc that appears after Item 18.

There is some divergence in the files.  In 43320, the items 7, 8, 9, 10, and 11 recycle a second time, whereas they recycle only once in 12067, and it also contains an extra latency time stamp that was not present in 12067.  Additionally, 12067 contains some extra data at the end of the file, specifically, the text of the MP3 file at the start of the file, an identifier string, a URL for Maps, and another identifier string.

File 12067 also had some extra data at the beginning that did not appear in 43320.

I also used Android Auto to test sending and receiving messages, and the file that represents that test is 22687.binarypb.  There were three sessions on 01/27/2019  The first session, which started at 14:16 (EST) went as follows:

Chat 1

About two minutes later at 14:16, a second session was started.  It went as follows:

Chat 2

About 3 minutes later (14:21 EST) I asked for directions using the same method as before (invocation via a button on my steering wheel).  The session went as follows:

Chat 3

The first thing I notice is there is a single binarypb file for 01/27/2019 (22687), even though there were three sessions.  Inspection of the file finds the 3-byte string, 0x5951EF, is present along with car_assistant string.  There is also a “yoda.”  See the orange, blue, and red boxes, respectively in Figure 24.  I carved from the yoda in Figure 24 to the end of the padding in Figure 25 (orange box).

FIgure 24

Figure 24.

Figure 25

Figure 25.

The following came out of my computer speakers:

“Smithfield Chicken and BBQ is 51 minutes from your location by car and light traffic.”

Now, this is interesting.  The first two sessions, which started at 14:16 and 14:18, did not include anything regarding directions.  The third session at 14:21 did involve directions.  I wonder if the fact that the three sessions were so close together that Google Assistant/Android just made one binarypb file to encompass all three sessions.  That would require more testing to confirm (or disprove) but is beyond the scope of this exercise and article.

Figures 26-1 and 26-2 shows the end of the MP3 data and some familiar data:  the Android Auto version information, /mic /mic, and a time stamp.  It also shows the QueryTriggerType and the webj gearhead* car_assistant gearhead string.  See the blue, orange, red, purple, respectively.  The time stamp here is 0x38CCC29068010000.  I converted to decimal (1548616911928), fired up DCode and got 01/27/2019 at 14:21:51 (EST).  This time is when I started the session in which I asked for directions.

Figure 26-1

Figure 26-1.

Figure 26-2

Figure 26-2.

Below that is some more familiar data.  The 5-byte string (0xBAF1C8F803) and 8-byte string (0x014C604080040200) appear (blue and green boxes in Figure 27-1), and there is the vocal input from my request for directions (that occurred roughly three minutes later).  There are also variations of the actual vocal input; each variation is designated by a letter (J, K, O, P, and M) (purple boxes).  After the variations, is the Android Auto version string, the /mic /mic string, and the same timestamp from before (orange and red boxes) (Figure 27-2).

Figure 27-1

Figure 27-1.

Figure 27-2

Figure 27-2.

The QueryTriggerType (red box) appears along with the webj gearhead* car_assistant J get me directions to the Smithfield Chicken & BBQ in Warsaw North Carolina gearhead (green box).  A BNDL appears (blue box), and then another time stamp (purple).  The byte string is 0x171DBC9068010000, and, in decimal is 1548616473879.  This converts to 01/27/2019 at 14:14 (EST), which is the time I hooked the phone to the car and started Android Auto.    See Figure 28.

Figure 28

Figure 28.

After that data is the velvet:query_state:search_result_id string, the accompanying 16-byte string ending in 0x12 (orange box), and a BNDL (blue box).  The 8-byte string (0x014C604080040200) appears (green box), my vocal input that started the first session (“read my newest text message” – purple box), and then a BNDL (blue box).  After that it is the Android Auto version, /mic /mic (yellow box), and a timestamp (red box).  See Figures 29-1 and 29-2.  The time stamp here is 0x0385BD9068010000, which is decimal 1548616566019.  When converted, it is 01/27/2019 at 14:16:06 (EST), which is the time of the first invocated session.

Figure 29-1

Figure 29-1.

Figure 29-2

Figure 29-2.

Also in Figure 29-2 is the QueryTriggerType and the webj gearhead* car_assistant (dark purple box) string.

Figure 30 has some new data in it.  A string appears that, while not completely similar, is something I had seen before.  It appears to be an identifier string:  GeQNOXLPoNc3n_QaG4J3QCw.  This is not 12 characters underscore 12 characters, but it is close.  Right after the identifier string is my vocal input “read my new text message.”  See the red box, and blue box, respectively in Figure 30.

Figure 30

Figure 30.

Figures 31-1 and 31-2 shows the two new text messages that were identified.  See the blue and red boxes.

Figure 31-1

Figure 31-1.

Figure 31-2

Figure 31-2.

Scrolling down a bit I find another identifier string:  eQNOXLPoNc3n_QaG4J3QCw (red box).  This identifier is the same as the first one, but without the leading “G.”  After this identifier is the velvet:query_state:search_result_id and the accompanying 16-byte string ending in 0x12 (orange box).  A BNDL appears at the end (green box).  See Figure 32.

Figure 32

Figure 32.

Next up is the 8-byte string (0x014C604080040200), and my next vocal input “read it.”  Just below my vocal input is the Android Auto version information, /mic /mic, and a time stamp.  Just below the time stamp is the QueryTriggerType and the webj gearhead* car_assistant gearhead strings (not pictured).  See the blue, orange, red, and purple boxes, respectively in Figures 33-1 and 33-2.  The time stamp here is 0xD796BD9068010000.  I converted it to decimal (1548616570583), fired up DCode and got 01/27/2019 at 14:16:10 (EST).  While I was not keeping exact time, this would seem to be when Google Assistant asked me whether or not I wanted to read the chat message from Josh Hickman.

Figure 33-1

Figure 33-1.

Figure 33-2

Figure 33-2.

There is another identifier string further down the file:  GeQNOXLPoNc3n_QaG4J3QCw and just below it my vocal input “read my newest text message.”  See the blue and red boxes, respectively, in Figure 34.  This is interesting.  Could it be that Google Assistant is associating this newest vocal input (“read it”) with the original request (“read my newest text message”) by way of the identifier string in order to know that the second request is related to the first?  Maybe.  This would definitely require some additional research if the case.

Figure 34

Figure 34.

Figures 35 and 36 show the text messages that were new when the request was made.

Figure 35

Figure 35.

Figure 36

Figure 36.

After some gobbly-goo, I found another identifier string:  gwNOXPm3FfKzggflxo7QDg (red box).  This format is completely different from the previous two I had seen.  Maybe this is an identifier for the vocal input “read it.”  Maybe it’s a transactional identifier…I am not sure.  See Figure 37.

Figure 37

Figure 37.

In Figure 37 you can also see the velvet:query_state:search_result_id and the 16-byte string ending in 0x12 (orange box), a BNDL, (blue box) the 8-byte string (green box), my next vocal input (purple box), and another BNDL.

Figure 38 shows familiar data:  Android Auto version, /mic /mic (green box), and a time stamp:  0x27BABD9068010000 (red box).  This converts to 1548616579623 in decimal, and 01/27/2019 at 14:16:19 in Unix Epoch Time.  As with the previous request, I wasn’t keeping exact time, but this would probably line up with when I said “Go on to the next one.”

Figure 38

Figure 38.

Figure 39 shows the QueryTriggerType string along with webj gearhead* car_assistant string.

Figure 39

Figure 39.

Figure 40 shows that identifier string again, and the vocal input that kicked off this session “read my newest text message.”  I am beginning to suspect this is actually some type of transactional identifier to associate “go on to the next one” with “read my newest text message.”

Figure 40

Figure 40.

Figures 41 and 42 show the new text messages.

Figure 41

Figure 41.

Figure 42

Figure 42.

There is another identifier string in Figure 43:  jwNOXMv_Ne_B_QbewpK4CQ (blue box).  This format is completely new compared to the previous ones.  Additionally, the velvet:query_state:search_result_id and the 16-byte string ending in 0x12 (orange box) appears, a BNDL (red box), along with the 8-byte string (green box) my next vocal input (purple box), and a BNDL.

Figure 43

Figure 43.

Figure 44 shows the Android Auto version, /mic /mic (blue box), and another time stamp (red box).  This time stamp is 0xC1ECBD9068010000, which converts to 1548616592577.  This is 01/27/2019 at 14:16:32 (EST).  This probably coincides with my vocal input “That’s it for now.” 

Figure 44

Figure 44.

Figure 45 has the QueryTriggerType and webj gearhead* car_assistant.

Figure 45

Figure 45.

Figure 46 shows a few things.  The first is the velvet:query_state:search_result_id and the 16-byte string ending in 0x12 (orange box).  The second thing is another identifier string, MgNOXNWTAtGp5wL03bHACg (blue box).  As before, this format does not match anything I have seen previously.

The third, and the most interesting part, is the start of the second session.  The only dividing line here is the velvet:query_state:search_result_id and the 16-byte string ending in 0x12, and BNDL (red box).  The green box is the 8-byte string, and the purple box contains my vocal input, “read my newest text messages.”  The purple boxes below are variations of what I said.

Figure 46

Figure 46.

Figure 47 shows the Android Auto version string, the /mic /mic string (blue box), and another time stamp (red box).  This time the stamp is 0x8C28C09068010000.  This converts to 1548616738956 in decimal, which is 01/27/2019 at 14:18:58 (EST) in Unix Epoch Time, which is the time I invoked Google Assistant for the second session.

Figure 47

Figure 47.

The next string that appears is the QueryTriggerType and webj gearhead* car_assistant strings.  See Figure 48.

Figure 48

Figure 48.

The next string is another identifier string.  This time, it is associated with my newest vocal input:  “read my newest text messages.” (blue box)  The string is GJgROXL2qNeSIggfk05CQCg (green box).  See Figure 49.

Figure 49

Figure 49.

Figures 50 and 51 show the text messages.

Figure 50

Figure 50.

Figure 51

Figure 51.

The next thing I see is a unique identifier string:  JgROXL2qNeSIggfk05CWCg (orange box).  This string is the same as the previous one (in Figure 49), but without the leading “G.”  This behavior is the same that I saw in the first session.  Beyond that there is the velvet:query_state:search_result_id and the 16-byte string ending in 0x12 (blue box), a BNDL (red box), the 8-byte string (green box), and my next vocal input, “hear it” (purple box), and another BNDL.  See Figure 52.

Figure 52

Figure 52.

Figure 53 shows the Android Auto version string, the /mic /mic string (blue box), and another time stamp (red box).  This time the stamp is 0x153AC09068010000.  This converts to 1548616743445 in decimal, which is 01/27/2019 at 14:19:03 (EST) in Unix Epoch Time, which would coincide with my vocal input “hear it.”

Figure 53

Figure 53.

The next string that appears is the QueryTriggerType and webj gearhead* car_assistant strings.  See Figure 54.

Figure 54

Figure 54.

Scrolling a bit finds an identifier string I have seen before:  GJgROXL2qNeSIggfk05CQCg (green box).  This is the first identifier seen in this session (the second one).  Just below it is the vocal input that started this session:  “read my newest text messages” (red box).  See Figure 55.

Figure 55

Figure 55.

Figures 56 and 57 show the messages that were new.

Figure 56

Figure 56.

Figure 57

Figure 57.

Figure 58 shows a pattern I have seen before.  First is another identifier string:  MAROXPPhAcvn_QaPpI24BA (orange box).  The second and third are velvet:query_state:search_result_id and the 16-byte string ending in 0x12 (blue box).  There is another BNDL (red box), the 8-byte string (green box), my next vocal input (purple box), “I’ll reply”, and another BNDL.  Also note the variations of what I said below my actual input (lower purple boxes).

Figure 58

Figure 58.

Figure 59 shows the Android Auto version string, the /mic /mic string (blue box), and another time stamp (red box).  This time the stamp is 0xD85CC09068010000.  This converts to 1548616752344 in decimal, which is 01/27/2019 at 14:19:12 (EST) in Unix Epoch Time, which would coincide with my vocal input “I’ll reply.”

Figure 59

Figure 59.

Figure 60 shows the next string that appears is the QueryTriggerType and webj gearhead* car_assistant strings.

Figure 60

Figure 60.

The next thing of interest is what is seen in Figure 61.  There is another identifier string, GPQROXMz3L6qOggfWpKeoCw (blue box).  This is not a string we have seen before.  Just below it is the vocal input that started this session (red box).

Figure 61

Figure 61.

I had to scroll quite a bit through some Klingon, but eventually I got to the area in Figure 62.  The red box shows another identifier string:  PQROXMz3L6qOggfWpKeoCw.  This is the same string that we saw in Figure 61, sans the leading “G.”  Again, this behavior is a pattern that we have seen in this particular file.  It causes my suspicion to grow that it is some type of transactional identifier that keeps vocal input grouped together.

Figure 62

Figure 62.

The second thing is velvet:query_state:search_result_id and the 16-byte string ending in 0x12 (blue box).  In Figure 63, there is another BNDL (red box), the 8-byte string (green box), and my next vocal input (the dictated message – purple box).  Also note the variations of what I said below my actual input (lower purple boxes).  Note that each variation is delineated by a character:  B, =, and >.

Figure 63

Figure 63.

Figure 64 shows the Android Auto version string, the /mic /mic string (blue box), and another time stamp (red box).  This time the stamp is 0x2192C09068010000.  This converts to 1548616765985 in decimal, which is 01/27/2019 at 14:19:25 (EST) in Unix Epoch Time, which would coincide with my dictation of a message to Google Assistant.

Figure 64

Figure 64.

Figure 65 shows the next string that appears is the QueryTriggerType and webj gearhead* car_assistant strings.

Figure 65

Figure 65.

The next thing of interest is what is seen in Figure 66.  There is an identifier string that we have seen before, GPQROXMz3L6qOggfWpKeoCw (blue box), and the initial vocal input that started this session (green box).  Again, I am beginning to think this is a method of keeping vocal inputs grouped within the same session.

Figure 66

Figure 66.

Scrolling through yet more Klingon, I find the area shown in Figure 67.  The blue box shows another identifier string:  RwROXPzPAcG5gge9s5n4DQ (red box).  This is a new identifier.  The velvet:query_state:search_result_id string and the 16-byte string ending in 0x12 (orange box) are also present.  There is another BNDL (blue box), the 8-byte string (green box), my next vocal input (purple box), “send it”, and another BNDL.

Figure 67

Figure 67.

Figure 68 shows the Android Auto version string, the /mic /mic string (blue box), and another time stamp (red box).  This time the stamp is 0x16BAC09068010000.  This converts to 1548616776214 in decimal, which is 01/27/2019 at 14:19:36 (EST) in Unix Epoch Time, which would coincide with my instructing Google Assistant to send the message I dictated.

Figure 68

Figure 68.

Figure 69 shows the next string that appears is the QueryTriggerType and webj gearhead* car_assistant strings.

Figure 69

Figure 69.

Figure 70 shows an identifier string, UQROXJ6aGKixggfh64qYDg (orange box), which is new.  Just below it is the velvet:query_state:search_result_id string and the 16-byte string ending in 0x12 (blue box) are also present.  There is another BNDL (red box), the 8-byte string (green box).  Below the 8-byte string is the vocal input that started the third session:  “get me directions to the Smithfield Chicken & BBQ in Warsaw North Carolina” (purple box).  Figure 71 shows the variations of my vocal input, which are identified by J, K, O, P, and M (purple boxes).  Below that is a BNDL.  This is the same data seen in Figure 27.

Figure 70

Figure 70.

Figure 71

Figure 71.

Figure 72 shows the Android Auto version string, the /mic /mic string (blue box), and another time stamp (red box).  This time the stamp is 0x38CCC29068010000.  This converts to 1548616911928 in decimal, which is 01/27/2019 at 14:21:51 (EST) in Unix Epoch Time.  This is the same time stamp seen in Figure 26.

Figure 72

Figure 72.

Figure 73 shows the next string that appears is the QueryTriggerType and webj gearhead* car_assistant strings.

Figure 73

Figure 73.

Scrolling down oh so slightly I find the text of the MP3 file at the beginning of the file (blue box), and the Google Maps URL for the location for which I had asked for directions (green box).  See Figure 74.  Figure 75 shows another identifier string, 1wROXOOFJc7j_Aa72aPAB (orange box).  After the identifier string is the velvet:query_state:search_result_id string.  Additionally, the string with the 16-byte string ending in 0x18 (green box), the 9-byte string (0x01B29CF4AE04120A10 – blue box), and the string and.gsa.d.ssc (red box) appear.

Figure 74

Figure 74.

Figure 75

Figure 75.

Comparisons

So, what is the final score?  If you’re still reading, let’s recap, and include what we found in the 22687.binarypb file.  The differences are in italics:

  1. The beginning of the file (the 3-byte 0x5951EF string and “car_assistant”).
  2. The MP3 audio at the beginning of the file which contains the last audio interaction prior to being sent to a different app (Maps).
  3. BNDL
  4. Android Auto Version along with the /mic /mic
  5. The date/time stamp of when Google Assistant is invoked, which appears just after some padding (0xFF). In the third file, 22687, the time stamp is the time for the third session.
  6. A 5-byte string (0xBAF1C8F803) that appears just before the vocal input appears the first time in a file. This string only appears here, and does not appear elsewhere. In the third file, 22687, this appeared before the first vocal input, which, as it turns out, is the vocal input that started the third session.
  7. An 8-byte string (0x014C604080040200) that appears just before the vocal input, regardless of where and how many times it appears within the file.
  8. Text of the vocal input.
  9. BNDL
  10. Android Auto Version along with the /mic /mic
  11. Another date/time stamp of when Google Assistant was invoked (same as the first).
  12. The string webj gearhead* car_assistant <my vocal input> gearhead (what I actually said). This item only appeared once in 22687 (the inquiry asking for directions).
  13. BNDL
  14. What I have decided to call a “latency time stamp,” although, it may indicate the last time any activity was done via Android Auto (including starting Android Auto) prior to the invocation of Google Assistant. In 22687, this only happened once.
  15. The velvet:query_state:search_result_id string appears along with the 16-byte string ending in 0x12.
  16. Items 7, 8, 9, 10, and 11 recycle.
  17. The velvet:query_state:search_result_id string appears along with the 16-byte string ending in 0x18, which appears at the end of the file.
  18. The 9-byte string 0x01B29CF4AE04120A10 after Item 17.
  19. The string gsa.d.ssc that appears after Item 18.

Conclusions

I feel comfortable enough at this point to draw a few conclusions based on my observations up to this point.

  1. Each binarypb file will start by telling you where the request is coming from (car_assistant).
  2. What is last chronologically is first in the binarypb file. Usually, this is Google Assistant’s response (MP3 file) to a vocal input just before being handed off to whatever service (e.g. Maps) you were trying to use.  The timestamp associated with this is also at the beginning of the file.
  3. A session can be broken down in to micro-sessions. I will call them vocal transactions.
  4. Vocal transactions have a visible line of demarcation by way of the 16-byte string ending in 0x12.
  5. A BNDL starts a vocal transaction, but also further divides the vocal transaction in to small chunks.
  6. The first vocal input in the binarypb file is marked by a 5-byte string: 0xBAF1C8F803, regardless of when, chronologically, it occurred in the session.
  7. Each vocal input is marked by an 8-byte string:   While the 5-byte string appears at the first in the binarypb file only, the 8-byte string appears just prior to each vocal input.
  8. When Google Assistant doesn’t think it understands you, it generates different variations of what you said…candidates…and then selects the one it thinks you said.  You can see thise in these files.
  9. In sessions where Google Assistant needs to keep things tidy, it will assign an identifier to vocal transactions. There does not appear to be any consistency (as far as I can tell) as to the format of these identifiers.
  10. The end of the final vocal transaction is marked by a 16-byte string ending in 0x18.

Figure 76 shows a visual version of the session files, over all, and Figure 77 shows the vocal transactions portion in more detail.

img_0075

Figure 76.

img_0074

Figure 77.

What’s Next?

So, there is definitely a format to these files, and I believe there are enough markers that someone could create a tool to parse them to make them easier to read and examine.  They contain what a device owner said…what they actually said…to their car/device.  This data could be extremely valuable to examiners and investigators, regardless of the venue in which they operate (civil or criminal).

Yes, this data could potentially be in a user’s Google Takeout data, and getting it that way would be slightly easier, although there would be a waiting period.  But, what if you do not have the authority to get said data?  This data could still be resident on the device and, potentially, available.

If I had scripting skills, I would try to see if something could be written to parse these files; however, alas, I do not.  I have minimal scripting skills.  Most of what I do is via Google and Cntrl/Cmd-C and Cntrl/Cmd-V.  If any reader can do this, and is willing, please let me know and I will post your product here and give full credit.  It would be awesome.

The second part of this post is forthcoming.  If you can’t wait, here’s a sneak peek:  there are some similarities…

Grab Your Glass of Milk! Android Oreo Image Now Available (8.x)

Now available for download is an Android Oreo (8.x) image, which was created using a LG Nexus 5X and a stock Android image. This image contains user populated data within the stock Android apps, along with user populated data within 25 non-stock apps.

As with the Android Nougat image, this one also includes robust documentation, a .ufd file, a list of hash values, and Takeout data from the ESPs who provided it.  This image is freely available to anyone who wants it for training, education, testing, or research.

Please note the image and related materials are hosted by Digital Corpora.  You can everthing here.