Results 1 to 17 of 17

Thread: SandForce HW compression

  1. #1
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,954
    Thanks
    359
    Thanked 332 Times in 131 Posts

    SandForce HW compression

    It's an interesting topic. SSDs with SandForce controllers use proprietary data compression to increase silicon lifespan and speed-up the bandwidth.

    My 240 GB Corsair Force GT recently died. I disassembled it and the thing looks straightforward - SandForce controller and bunch of ICs. Anyway, be really careful connecting a SATA cable to such SSDs!

    Replaced it with 240 GB Corsair Force GS (85000 IOPS -> 90000 IOPS)

    I think it would be interesting to reverse engineer this compression algorithm. My guesses, SandForce HW compression is similar to the NTFS' LZSS.
    Attached Thumbnails Attached Thumbnails Click image for larger version. 

Name:	IMG_0642.jpg 
Views:	435 
Size:	1.15 MB 
ID:	2199   Click image for larger version. 

Name:	IMG_0646.jpg 
Views:	368 
Size:	1.07 MB 
ID:	2200   Click image for larger version. 

Name:	IMG_0647.jpg 
Views:	310 
Size:	1.06 MB 
ID:	2201   Click image for larger version. 

Name:	IMG_0649.jpg 
Views:	511 
Size:	1.08 MB 
ID:	2202  

  2. #2
    Member
    Join Date
    Jun 2009
    Location
    Kraków, Poland
    Posts
    1,471
    Thanks
    26
    Thanked 120 Times in 94 Posts
    A little OT:
    My 240 GB Corsair Force GT recently died.
    How long it lived?

  3. #3
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,954
    Thanks
    359
    Thanked 332 Times in 131 Posts
    About 1.5 years. This broken connector is not the only issue. Over the last few months I had some data loss issues - compression testing is a poison for such devices with such a limited write capability. Of course, I'm using RAM disk in most cases, but...
    So, keep it in mind and never keep important data on SSD - do a backup on removable HDD and/or CD. Still SSD is a fastest sh1t!

  4. #4
    Member
    Join Date
    Jan 2013
    Location
    Australia
    Posts
    5
    Thanks
    0
    Thanked 0 Times in 0 Posts

    SandFarce

    Quote Originally Posted by encode View Post
    About 1.5 years. This broken connector is not the only issue. Over the last few months I had some data loss issues - compression testing is a poison for such devices with such a limited write capability. Of course, I'm using RAM disk in most cases, but...
    So, keep it in mind and never keep important data on SSD - do a backup on removable HDD and/or CD. Still SSD is a fastest sh1t!
    Encode:

    A few weeks back I posted a thread here asking about HW compression, guess what I am looking into?

    The SandForce controllers are an incorrectly engineered heap of shit(tm) and shouldn't be trusted with anyone's data, we have had many failures of these (both SF-1xxx and SF-2xxx series) and have lost significant amounts of data.

    If you still have the drive in your possesion, can I suggest you dump the status logs (via serial UART from the debug port) and put them up? I'd like to see if your failure mode matches those of our drives (also 240GB variants with SF-228x controllers, albeit ours were Intel 520 SSDs).

    On your first attached pics, the 4 pins missing a header is the SIO debug port (UART at 3.3v) so it should dump text out at 115k2 @ 8N1 with the failure logs when powered up (it doesn't need to have the SATA connected, just power) to do so.

    Additional (edit):

    A couple of important questions to ask you:

    1/. Were you using the drive on a 6GIG or 3GIG SATA port?

    2/. Was the drives host port configured for AHCI, IDE or RAID?

    3/. Did you have any S3/Sleep/Hibernate modes enabled?

    Luntik.
    Last edited by Luntik; 22nd February 2013 at 04:30. Reason: Added questions.

  5. #5
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,954
    Thanks
    359
    Thanked 332 Times in 131 Posts
    1. 6G SATA (SATA3)
    2. AHCI
    3. No S3/Sleep/Hibernation. Also I've disabled Paging File and the Browser HDD cache. No reason for such obsolete things with 16 GB of RAM.

    Is it possible to get that dump with no specialized hardware/readers?

  6. #6
    Member
    Join Date
    Jan 2013
    Location
    Australia
    Posts
    5
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by encode View Post
    1. 6G SATA (SATA3)
    2. AHCI
    3. No S3/Sleep/Hibernation. Also I've disabled Paging File and the Browser HDD cache. No reason for such obsolete things with 16 GB of RAM.

    Is it possible to get that dump with no specialized hardware/readers?
    Encode:

    Ok, your experience matches ours, except for the fact that you don't seem to be using hibernate/sleep modes. The majority of our failures have all been on 6G ports (not to say the problem isn't prevalent on 3G ports, but it seems that the 6G ports bring about the failure mode much quicker).

    With respect to the UART, the easiest way is to use an FTDI breakut cable or board (place likes Sparkfun sell them, or Digikey, etc) just ensure you get the 3.3v one. All that is required is to hook up GND, Vcc, RX and TX to the debug port and capture the input on the devices POR.

    Can I ask what the drive was being used for? Ours have all been the Windows7 boot drives in quite powerful workstations (6 core CPUs w/ 64GB RAM). I'm guessing from what you mentioned that you performed a lot of compressed data writes to the drives which would have increased write amplification to their FTL - we work with a lot of encrypted data so we effectively have a similar issue so there is a correlation there.

    Can you recall any instances of where the system has locked up and you've had to either reset or power off without issuing a normal shutdown? Have you had any power failures where the system didn't correctly shutdown?

    We're pretty sure that the issue is down to the drives not properly completing long writes and trashing their internal states on subsequent power on events. In the last two failures we had, the machines appeared to freeze upon wake-up (ie: resume from sleep) and the SATA bus was held high as the device put itself in panic mode and offline. So we think the failure was actually triggered on the previous power-down event and only becomes evident on the subsequent reboot/wake-up.

    We're currently delving into the FTL to examine their real-time encryption and compression operations in an attempt to get back some of our critical data held on two of the drives. We're currently getting a RAW dump of the drives from underneath the FTL, so we can further examine their encryption and compression layers.

    Luntik.

  7. #7
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,954
    Thanks
    359
    Thanked 332 Times in 131 Posts
    It was a system drive - Windows 7, some torrent downloads and games. And of course all my compression related stuff - Visual Studio projects, test files. I've just realized that each new compile is a new disk write. I bet I could kill an SSD in one day just by running my optimizer on it - that's why I'm using RAM-drives and in-memory compression tests.
    I have never experienced any power failures or system locks. Even worse, data loss looked like files filled with zeroes. Proper size, but file contents are all zero bytes. This is heavy...

  8. #8
    Member
    Join Date
    Nov 2012
    Location
    Bangalore
    Posts
    114
    Thanks
    9
    Thanked 37 Times in 22 Posts
    In addition to compression it uses block-level deduplication. These Sandforce controllers exhibit multiple problems:
    http://erratasec.blogspot.in/2012/07...ntrollers.html
    Their compression performs poorly on incompressible data.

    In addition doing transparent dedupe causes problems for filesystems that try to store redundant copies of critical metadata for recovery purposes:
    http://queue.acm.org/detail.cfm?id=1985003

    Deduplication without the filesystem's knowledge will collapse multiple metadata copies into a single one thereby defeating the purpose.

  9. #9
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,612
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Quote Originally Posted by moinakg View Post
    In addition doing transparent dedupe causes problems for filesystems that try to store redundant copies of critical metadata for recovery purposes:
    http://queue.acm.org/detail.cfm?id=1985003

    Deduplication without the filesystem's knowledge will collapse multiple metadata copies into a single one thereby defeating the purpose.
    According to Sandforce, they use internal RAID-like scheme to spread data to multiple chips, so a single failure won't harm your data.
    Source:
    https://storagemojo.com/2011/06/27/d...of-good-thing/

  10. #10
    Member
    Join Date
    Jan 2013
    Location
    Australia
    Posts
    5
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by m^2 View Post
    According to Sandforce, they use internal RAID-like scheme to spread data to multiple chips, so a single failure won't harm your data.
    Source:
    https://storagemojo.com/2011/06/27/d...of-good-thing/

    That's absolute rubbish - I suggest you ignore all of the PR bullshit, the old adage of "don't believe everything you read" still holds true.

    We have now RE'd enough of their SATA device controller and their FTL to the point where we can expose both significant flaws and outright lies with respect to their "AES encryption" engine and their "de-duplication" process.

    Luntik.

  11. #11
    Member
    Join Date
    Apr 2012
    Location
    Stuttgart
    Posts
    437
    Thanks
    1
    Thanked 96 Times in 57 Posts
    Quote Originally Posted by moinakg View Post
    Their compression performs poorly on incompressible data.
    Isn't that the point? I'm really curious to see a compressor that performs "well" on incompressible data.

  12. #12
    Member
    Join Date
    Oct 2009
    Location
    usa
    Posts
    56
    Thanks
    1
    Thanked 9 Times in 6 Posts
    Looks like SandForce-based SSDs are definitely to be avoided for those of us who would mostly use these drives for data which is mostly incompressible. I would mainly use the such a drive to author blu-rays, so what would be the best controller / chipset for my usage? The fast data-transfer rate is what will make this process much faster. I do not want or need any compression. Suggestions?

  13. #13
    Member Bloax's Avatar
    Join Date
    Feb 2013
    Location
    Dreamland
    Posts
    52
    Thanks
    11
    Thanked 2 Times in 2 Posts
    Quote Originally Posted by thorfdbg View Post
    I'm really curious to see a compressor that performs "well" on incompressible data.
    Not slowing down to a crawl processing incompressible data is by definition a "good" performance on it. :v
    • And judging by how it's said that SandForce SSDs perform worse than others on such data...

  14. #14
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    zyzzle, consider 4 hdds in raid0. it have even better linear r/w speeds at the start of disk and plenty of room

    of ssds, new ocz vector has 500 mb/s read and write speeds, ocz vertex4 is a bit slower but cheaper. i recommend plextor 5s that is reliable, pretty fast and pretty cheap
    Last edited by Bulat Ziganshin; 25th February 2013 at 10:37.

  15. #15
    Tester
    Black_Fox's Avatar
    Join Date
    May 2008
    Location
    [CZE] Czechia
    Posts
    471
    Thanks
    26
    Thanked 9 Times in 8 Posts
    Samsung (830/840) and Crucial (m4) are also brands worth recommending, the 830/m4 should be pretty cheap these days. Intel should theoretically be fine too, but I have no SSDs from them so I can't say.
    I am... Black_Fox... my discontinued benchmark
    "No one involved in computers would ever say that a certain amount of memory is enough for all time? I keep bumping into that silly quotation attributed to me that says 640K of memory is enough. There's never a citation; the quotation just floats like a rumor, repeated again and again." -- Bill Gates

  16. #16
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 659 Times in 354 Posts
    samsung is just rare here in russia, although i know it have good speed and recommendations. M4 has the same price as 5s, but a bit less linear speeds. Intel has two sandforce-based drives. they are reliable, but linear speeds on incompressible data are not that high

  17. #17
    Member
    Join Date
    Feb 2013
    Location
    San Diego
    Posts
    1,057
    Thanks
    54
    Thanked 71 Times in 55 Posts
    I have had a Samsung 830 for close to 2 years and I have not had any problems with it. For a long time, I was an avid reader of anandtech.com, which has been on top of the SSD story since the very beginning, and they have basically singled out Samsung and Intel as the top manufacturers for reliability. Samsung does not use Sandforce controllers, and Intel started using Sandforce recently. But Intel spent a solid year debugging the firmware before bringing their first SF-based drive to market. This extra-debugged firmware is exclusive to Intel drives.

    When I started reading about stability problems with the SandForce 22xx controllers, it got me thinking. SSDs undoubtedly need pretty complex firmware just to get around the limitations of flash. It took a few years and a few generations of controllers for companies just to solve all the puzzles around inconsistent latencies and write-amplification, and any controller+firmware these days has to achieve that level of sophistication just to be a credible product. The number of lines of code in firmware has got to be well beyond what was good enough in a HDD. All of that code has to be just about perfect, too, because SSDs don't just crash; they brick; and every screwed up pointer is going to lose some data. The threat of data loss from buggy firmware is probably orders of magnitude higher than from decaying flash. Flash goes bad at a predictable and measurable rate, and what risk exists can be mitigated with replication. HDDs somehow manage to reliably store data on swiftly rotating magnetic platters, which sounds ridiculously harder.

    It's obvious that SandForce has the most complex firmware of any SSD manufacturer. Samsung uses lots of DRAM for buffering, which is more of a brute-force approach to the kinds of things SF does by being clever in firmware. But Samsung makes less glitchy SSDs.

Similar Threads

  1. SandForce SSD Compression
    By Cyan in forum Data Compression
    Replies: 7
    Last Post: 11th February 2012, 22:18

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •