Ponder The Bits

Musings and confusings. All things DFIR.

Category: Incident Response

Quick(er) Mounting and Dismounting of LVM’s on Forensic Images

I recently came across Int’l Man of Leisure’s blog posts here and here on “Mounting and Imaging Logical Volume Manager (LVM2)”. He does a great job of defining the problem statement (dealing with LVM’s in their various image formats in a DFIR investigation) and how to work through getting a set of logical images back into their intended LVM layout for appropriate mounting and analysis.

IMOL begins by going through the background of LVM, what it is, and how to install it to prepare your system for dealing with LVM’s. Once prepared, IMOL begins by presenting a set of two VMDK images that must be merged or “stitched together” in order to be interpreted and parsed by the Linux LVM. However, VMDK files are not something natively readable/mountable by a linux system. So, before we can even begin stitching these back together, these VMDK files must be converted into something natively readable by the system, such as a raw image/block device. In IMOL’s testing, he found that FTK Imager was one of a few tools that was able to read the VMDK files in order to convert (image) them to raw files. He then used FTK Imager to image the VMDK files into respective Raw DD format files for continued use. However, here is where I would like to branch into my process for mounting LVM’s that completely eliminates the need for converting an image to raw using a tool called “QEMU”, and thus saving you (potentially) hours of time.

We all know when dealing with forensic imaging/conversion that even the slightest hiccup can render an entire image useless and long-spent time wasted. The less time we spend imaging/converting, the faster we can get to analysis and toward our goals for the investigation. Enter QEMU, specifically “qemu-nbd”. I could go into a lot of detail of all the types of images it can convert and how useful it can be in various capacities (in fact, I may do another blog post about it). However, for this post, I will just stick to specifically how you can use it to perform on-the-fly image format translation (in real-time) between various formats – no need to spend time converting to another image file.

QEMU has a utility called “qemu-nbd” (nbd stands for Network Block Device) that essentially performs real-time translation betwixt (I have waited so long to use that word in a serious tone) various image formats. It’s as easy as the following:

Ensure you have an available NBD
# ls -l /dev/nbd*

If no device files (or not enough) exist as /dev/nbd*, create as many as needed
# for i in {0..<z>}; do mknod /dev/nbd$z b 43 $z; done
*Where <z> is the number of devices you need, minus one

Mount the image file
# qemu-nbd -r -c /dev/nbd<x> image.<extension>
* -r: read-only
* -c: connect image file to NBD device
*Where <x> is the appropriate block device number (typically starting at 0) and <extension> is a supported QEMU Image Type (raw, cloop, cow, qcow, qcow2, vmdk, vdi, vhdx, vpc)

Note that this will need to be done for each image that is a part of the LV group. For example, if there are 3 different VMDK files that together comprise one or more LV groups, you would do the following (ensuring the associated /dev/nbd devices have already been created before issuing the below commands):

# qemu-nbd -r -c /dev/nbd0 image_0.vmdk
# qemu-nbd -r -c /dev/nbd1 image_1.vmdk
# qemu-nbd -r -c /dev/nbd2 image_2.vmdk

That’s it. Each /dev/nbd<x> is immediately translated and available as a raw block device to be queried/mounted just as if it were a raw image to begin with. Pretty awesome, huh?

Now, if you were lucky enough to start with raw/DD images, you don’t need to perform any of the above. Instead, you can just skip to the below instructions for mounting and mapping the LVM’s.

At this point, my process to identify and load the LVM(s) mostly mirrors that as described by IMOL, with a few subtle differences. I won’t go into great detail of it all as IMOL gives great descriptions of each step in his walk-through. However, I will lay out my commands below for those who are looking for an easy copy/paste method to stick into their cheat sheets.

Keep in mind that the order of the below commands is critical to successful mounting of LVM’s.

Load the LVM module if not already loaded
# modprobe dm_mod

Ensure you have enough available loopback devices (one for each nbd device)
# ls -l /dev/loop*

If not enough loopback devices exist, create as many as needed
# for i in {0..<z>}; do mknod /dev/loop$z b 7 $z; done
*Where <z> is the number of devices you need, minus one

Set up a loopback device for each image that is part of the LV group
# losetup -r [-o <byte_offset>] -f [/dev/loop<x>] /dev/nbd<x>

Read partition tables from each loopback device to create mappings
# kpartx -a -v /dev/loop<x>

List the available Physical Volume Groups (VG’s)
# pvdisplay

List the available Logical Volume Groups (VG’s)
# lvdisplay

(Optional) If not listed, re-scan the mounted volumes to identify the associated VG’s
# pvscan
# lvscan
# vgscan

Activate the appropriate VG’s
# vgchange -a y <VG>
** The recombined LVM volume(s) will now be available at /dev/mapper/<VolumeGroup>-<VolumeName>

Mount the LVM Volume(s)
# mount [options] /dev/mapper/<VolumeGroup>-<VolumeName> /mnt/point

Congratulations. You (should) now have filesystem access to the given LVM(s)!

== Sidebar ==

Interested in why we use the given numbers of “43” and “7” for the mknod command?

The mknod command is structured like the following: mknod <device> <type> <major_#> <minor_#>

For our uses, we are creating device files of type “b” (block device), with major #’s of “43” (nbd) and “7” (loopback). The major number tells the system what type of device to expect and operate as. For a list of devices that your system is aware of and can dynamically assign when a major number is not specified, check out your /proc/devices file. IBM does a rather good job of explaining it all here. The minor number is more for reference and thus should probably, as best practice, reference/relate to the device number as well. Though, there is nothing requiring it to be that way.

For further information about the mknod command structure, just check out the man page.

== /Sidebar ==

Once you’re done with the images, the next logical step is to dismount the images which can at times become unnecessarily and illogically troublesome. To properly dismount LVM’s, perform the following steps (again, order is critical here!):

Dismount each mounted filesystem
# umount /mnt/point

De-activate each activated Volume Group
# vgchange -a n <VG>

Remove partition mappings for each loop device
# kpartx -d -v /dev/loop<x>

Remove each loopback device
# losetup -d /dev/loop<x>

(Optional) Force remove an LVM
# dmsetup remove -f <VG>

Keep in mind the dmsetup command above is considered deprecated and not suggested for use. However, I provide it as I have had to use it at times in the past when a VG simply would not detach using the appropriate commands. That said, if all else fails, reboot 🙂

Hopefully, this post will help with the often convoluted process of mounting LVM’s, especially when split across multiple images/devices.


The Importance of Incident Scoping/Assessment

In consulting, all engagements begin with what we refer to as “scoping” in order to, at a very high level, determine if/how we may be able to help a client. Sure, they can sometimes be arduous or monotonous and often involve a comedy of conference call errors, but they are absolutely CRITICAL to the success of the engagement. So critical, in fact, that I wanted to take some purposeful time to address this often-overlooked, admittedly very unsexy, but nonetheless integral piece of performing effective DFIR.

Though I’m approaching this from a consulting point of view, this is in no way unique nor applicable only to consulting. So, don’t tune out now just because, “I don’t have clients.” You do, actually, whether they are external entities or people within your own organization. Each of us has “clients” in various forms that rely (often heavily) on us as DFIR professionals to ask the right questions and respond with the appropriate guidance and/or actions to mitigate threats and protect them from harm. As such, every DFIR professional will find themselves in situations where they must first comprehensively understand the problem(s) and issue(s) at hand before it can be effectively addressed in response and analysis. In fact, this is often where many of us find ourselves at the onset of an alert or suspected compromise that requires a well-formulated and concerted DFIR response. Though it can be very difficult to take step back and perform a high level assessment instead of diving right into action (especially when you may have external entities and/or higher level management wanting answers LIKE YESTERDAY), doing so will pay you back ten-fold.

At a lower level, scoping (or an initial assessment) must be performed as a due diligence effort to acquire all appropriate/pertinent information, addressing a variety of aspects that may affect the success, efficacy, or efficiency of the investigation. From my private sector experience dealing with a wide array of clients from all types of verticals, industries, and organizational sizes, I’ve attempted to unjustly distill the scoping topic and question set down a few (ok, a few more than a few) important bullets provided below.

However, do keep in mind that this is not intended to be all-inclusive set of questions to ask by any means. Rather, the below set of topics and specific questions are intended to serve as a solid baseline in facilitating comprehensive response and should be augmented or modified as needed. In addition, the below questions are from the perspective of me/us asking an external entity (i.e. client). So, feel free to change/replace the pronouns as well as instances of “the client” as needed for your use and application.

  • Purpose: Understand the background of the situation
    • How did we (the client/organization) get here?
    • When and how did they first notice any sort of issue?
    • Can they think of any legitimate (non-malicious) causes for the current issue?
    • Have there been any other anomalies either previous to or during the timeframe of concern that may be related to the current issue?
  • Purpose: Understand what responsive actions have been performed to date
    • Are the systems still running or have they been powered down?
    • Have firewall blocks been put in place?
    • (Tons more questions I ask…)
    • Have any external entities been notified (particularly if under certain regulations)?
  • Purpose: Understand the set of artifacts available to assist us in our response
    • What type(s) of machines are involved (workstations/servers, OS)?
    • Are they virtual or physical?
    • Where are they hosted, and do they span disparate physical locations?
    • What specific logging is enabled (host, network, and appliance)?
    • …and which of the above are actually collected and available?
    • …and do the available logs contain the right (useful) content?
      • Let’s just say it is not uncommon to get “VPN logs” that simply show syslog interface up/down, “Web Logs” that contain only the load balancer as the “client-ip”, or “DNS logs” that aren’t logging query/response and logging only the IP of the DNS server as the “client”.
    • …and do the available logs span the time frame(s) of concern/interest?
  • Purpose: Identify the goals (in priority order) for the engagement
    • Ask the question, “What would a successful engagement look like to you?”
    • Is the client interested in simply getting back to business?
    • Would they like a root cause analysis (how exactly did this get in/past their defenses)?
      • If so, are they looking to specifically prove/disprove something?
    • Are they simply looking to check a box (fulfilling some sort of requirement)?
      • Often this will not be stated, but you can determine it by asking certain questions and gauging certain entities’ investment in the response.
    • What would they like us to do, specifically, in priority order?
      • Now, we don’t take this verbatim and slap it into a SoW, but we do use it to guide our response as best as possible to achieve their priority goals in tandem with an appropriate, comprehensive, and best practice response.

Though it is not uncommon for a situation to merit even further questioning and level of excruciating detail, depending on complexity, the above set of topics are what I would consider the minimum requirements for a comprehensive initial assessment. Suffice to say that misunderstandings or simple lack of required information in any one of these areas can lead to a variety of undesirable consequences, ranging from that of a simply sub-optimal outcome to that of a completely disastrous one for both the company and the client.

The DFIR industry is a “pay now or pay later” industry, and this is no exception. As such, I highly encourage all of us to be purposeful in spending our time performing due diligence on the front end instead of paying the consequences of not doing so throughout our investigations.


Powered by WordPress & Theme by Anders Norén