Configure Oracle Exadata Write Back Flash Cache

VitalSoftTechBlog

VitalSoftTechBlog

In addition to improving read I/Os, Oracle Exadata Write back flash cache also provides the ability to cache write I/Os directly to PCI flash.

Exadata storage software version 11.2.3.2.1 is the minimum version required to use write back flash cache. Grid infrastructure and database homes must run 11.2.0.3.9 or later to use with Write-back Smart Flash Cache. Database homes running 11.2.0.2 must install Patch 17342825.

As long as the minimum software requirements are met, any Exadata hardware with flashcache (V2 and later) can take advantage of this feature. You should consider using the write back flash cache, if your application writes intensively, significant waits for “free buffer waits” or high IO times.

Write-back flash cache significantly improves the write intensive operations because writing to flash cache is faster than writing to Hard disks. Write-back flash cache (powered by battery) is persistent upon reboots.

How to check if write back flash cache is enabled?

[root@dm01db01 ~]# dcli -g ~/cell_group -l root cellcli -e “list cell attributes flashcachemode”
dm01cel01: WriteThrough
dm01cel02: WriteThrough
dm01cel03: WriteThrough

This shows that Write back flash cache is NOT ENABLED.

Steps to enable the write back flash cache
These steps are carried out on an Exadata X4 Quater RACK (2 Coumpute node, 3 Storage cells & 1 Infiniband Switch)

Before performing the below steps, you should perform some prerequisite checks to ensure that all cells are in the proper state.
Please perform the following check as root from one of the compute nodes:

Prerequisites
– Check all grdidksk asmdeactivationoutcome and asmmodestatus to ensure that all griddisks on all cells are “Yes” and “ONLINE” respectively.

[root@dm01db01 ~]# dcli -g cell_group -l root cellcli -e list griddisk attributes asmdeactivationoutcome, asmmodestatus
dm01cel01: Yes ONLINE
dm01cel01: Yes ONLINE
dm01cel01: Yes ONLINE
dm01cel01: Yes ONLINE
dm01cel01: Yes ONLINE


dm01cel03: Yes ONLINE
dm01cel03: Yes ONLINE

– Check that all of the flashcache are in the “normal” state and that no flash disks are in a degraded or critical state:

[root@dm01db01 ~]# dcli -g cell_group -l root cellcli -e list flashcache detail

dm01cel01: name: dm01cel01_FLASHCACHE
dm01cel01: cellDisk: FD_10_dm01cel01,FD_14_dm01cel01,FD_01_dm01cel01,FD_15_dm01cel01,FD_08_dm01cel01,FD_03_dm01cel01,FD_04_dm01cel01,FD_06_dm01cel01,FD_09_oracloudceladm01,FD_00_dm01cel01,FD_02_dm01cel01,FD_07_dm01cel01,FD_13_dm01cel01,FD_05_dm01cel01,FD_12_dm01cel01,FD_11_dm01cel01
dm01cel01: creationTime: 2014-05-15T12:27:33+05:30
dm01cel01: degradedCelldisks:
dm01cel01: effectiveCacheSize: 2978.75G
dm01cel01: id: 35071680-cbfe-47be-b111-414938ed9b79
dm01cel01: size: 2978.75G
dm01cel01: status: normal

dm01cel02: name: dm01cel02_FLASHCACHE
dm01cel02: cellDisk: FD_12_dm01cel02,FD_02_dm01cel02,FD_13_dm01cel02,FD_15_dm01cel02,FD_04_dm01cel02,FD_01_dm01cel02,FD_07_dm01cel02,FD_03_dm01cel02,FD_00_oracloudceladm02,FD_14_dm01cel02,FD_08_dm01cel02,FD_10_dm01cel02,FD_05_dm01cel02,FD_09_dm01cel02,FD_06_dm01cel02,FD_11_dm01cel02
dm01cel02: creationTime: 2014-05-15T12:27:56+05:30
dm01cel02: degradedCelldisks:
dm01cel02: effectiveCacheSize: 2978.75G
dm01cel02: id: 768e04cd-5ce0-4b40-a79b-e2eac72efbe8
dm01cel02: size: 2978.75G
dm01cel02: status: normal

dm01cel03: name: dm01cel03_FLASHCACHE
dm01cel03: cellDisk: FD_11_dm01cel03,FD_10_dm01cel03,FD_07_dm01cel03,FD_02_dm01cel03,FD_06_dm01cel03,FD_15_dm01cel03,FD_00_dm01cel03,FD_08_dm01cel03,FD_01_oracloudceladm03,FD_03_dm01cel03,FD_09_dm01cel03,FD_05_dm01cel03,FD_14_dm01cel03,FD_12_dm01cel03,FD_04_dm01cel03,FD_13_dm01cel03
dm01cel03: creationTime: 2014-05-15T12:27:07+05:30
dm01cel03: degradedCelldisks:
dm01cel03: effectiveCacheSize: 2978.75G
dm01cel03: id: 3c16098f-d9fe-4d12-86e7-adca8fcc6446
dm01cel03: size: 2978.75G
dm01cel03: status: normal

Enable Write Back Flash Cache using a ROLLING method

In this method write-back flashcache is enabled, one cell at a time. The RDBMS & ASM instance will remain up and running.

– Log onto the first cell that you wish to enable write-back FlashCache

1. Drop the flash cache on that cell

[root@dm01cel01 ~]# cellcli -e drop flashcache
Flash cache dm01cel01_FLASHCACHE successfully dropped

2. Check if ASM will be OK if the grid disks go OFFLINE. The following command should return ‘Yes’ for the grid disks being listed:

[root@dm01cel01 ~]# cellcli -e list griddisk attributes name,asmmodestatus,asmdeactivationoutcome
DATAC1_CD_00_dm01cel01 ONLINE Yes
DATAC1_CD_01_dm01cel01 ONLINE Yes
DATAC1_CD_02_dm01cel01 ONLINE Yes
..
..
DBFS_DG_CD_07_dm01cel01 ONLINE Yes
DBFS_DG_CD_08_dm01cel01 ONLINE Yes
DBFS_DG_CD_09_dm01cel01 ONLINE Yes
DBFS_DG_CD_10_dm01cel01 ONLINE Yes
DBFS_DG_CD_11_dm01cel01 ONLINE Yes


RECOC1_CD_10_dm01cel01 ONLINE Yes
RECOC1_CD_11_dm01cel01 ONLINE Yes

3. Inactivate the griddisk on the cell

[root@dm01cel01 ~]# cellcli -e alter griddisk all inactive
GridDisk DATAC1_CD_00_dm01cel01 successfully altered
GridDisk DATAC1_CD_01_dm01cel01 successfully altered
GridDisk DATAC1_CD_02_dm01cel01 successfully altered
GridDisk DATAC1_CD_03_dm01cel01 successfully altered


GridDisk DBFS_DG_CD_02_dm01cel01 successfully altered
GridDisk DBFS_DG_CD_03_dm01cel01 successfully altered
GridDisk DBFS_DG_CD_04_dm01cel01 successfully altered


GridDisk RECOC1_CD_00_dm01cel01 successfully altered
GridDisk RECOC1_CD_01_dm01cel01 successfully altered
GridDisk RECOC1_CD_02_dm01cel01 successfully altered

4. Shut down cellsrv service

[root@dm01cel01 ~]# cellcli -e alter cell shutdown services cellsrv

Stopping CELLSRV services…
The SHUTDOWN of CELLSRV services was successful.

5. Set the cell flashcache mode to writeback

[root@dm01cel01 ~]# cellcli -e “alter cell flashCacheMode=writeback”
Cell dm01cel01 successfully altered

6. Restart the cellsrv service

[root@dm01cel01 ~]# cellcli -e alter cell startup services cellsrv

Starting CELLSRV services…
The STARTUP of CELLSRV services was successful.

7. Reactivate the griddisks on the cell

[root@dm01cel01 ~]# cellcli -e alter griddisk all active
GridDisk DATAC1_CD_00_dm01cel01 successfully altered
GridDisk DATAC1_CD_01_dm01cel01 successfully altered
GridDisk DATAC1_CD_02_dm01cel01 successfully altered
..
..
GridDisk DBFS_DG_CD_02_dm01cel01 successfully altered
GridDisk DBFS_DG_CD_03_dm01cel01 successfully altered
GridDisk DBFS_DG_CD_04_dm01cel01 successfully altered
..
..
GridDisk RECOC1_CD_00_dm01cel01 successfully altered
GridDisk RECOC1_CD_01_dm01cel01 successfully altered
GridDisk RECOC1_CD_02_dm01cel01 successfully altered
..
..

8. Verify all grid disks have been successfully put online using the following command:

[root@dm01cel01 ~]# cellcli -e list griddisk attributes name, asmmodestatus
DATAC1_CD_00_dm01cel01 ONLINE
DATAC1_CD_01_dm01cel01 ONLINE
DATAC1_CD_02_dm01cel01 ONLINE
..
..
DBFS_DG_CD_02_dm01cel01 SYNCING
DBFS_DG_CD_03_dm01cel01 SYNCING
DBFS_DG_CD_04_dm01cel01 SYNCING
..
..
RECOC1_CD_00_dm01cel01 OFFLINE
RECOC1_CD_01_dm01cel01 OFFLINE
..
..

Wait few seconds and check the status again.

[root@dm01cel01 ~]# cellcli -e list griddisk attributes name, asmmodestatus
DATAC1_CD_00_dm01cel01 ONLINE
DATAC1_CD_01_dm01cel01 ONLINE
DATAC1_CD_02_dm01cel01 ONLINE
..
..
DBFS_DG_CD_02_dm01cel01 ONLINE
DBFS_DG_CD_03_dm01cel01 ONLINE
DBFS_DG_CD_04_dm01cel01 ONLINE
..
..
RECOC1_CD_00_dm01cel01 SYNCING
RECOC1_CD_01_dm01cel01 SYNCING
RECOC1_CD_02_dm01cel01 SYNCING
..
..

Wait few more seconds and check the status again.

[root@dm01cel01 ~]# cellcli -e list griddisk attributes name, asmmodestatus
DATAC1_CD_00_dm01cel01 ONLINE
DATAC1_CD_01_dm01cel01 ONLINE
DATAC1_CD_02_dm01cel01 ONLINE
DATAC1_CD_03_dm01cel01 ONLINE
DATAC1_CD_04_dm01cel01 ONLINE
DATAC1_CD_05_dm01cel01 ONLINE
DATAC1_CD_06_dm01cel01 ONLINE
DATAC1_CD_07_dm01cel01 ONLINE
DATAC1_CD_08_dm01cel01 ONLINE
DATAC1_CD_09_dm01cel01 ONLINE
DATAC1_CD_10_dm01cel01 ONLINE
DATAC1_CD_11_dm01cel01 ONLINE
DBFS_DG_CD_02_dm01cel01 ONLINE
DBFS_DG_CD_03_dm01cel01 ONLINE
DBFS_DG_CD_04_dm01cel01 ONLINE
DBFS_DG_CD_05_dm01cel01 ONLINE
DBFS_DG_CD_06_dm01cel01 ONLINE
DBFS_DG_CD_07_dm01cel01 ONLINE
DBFS_DG_CD_08_dm01cel01 ONLINE
DBFS_DG_CD_09_dm01cel01 ONLINE
DBFS_DG_CD_10_dm01cel01 ONLINE
DBFS_DG_CD_11_dm01cel01 ONLINE
RECOC1_CD_00_dm01cel01 ONLINE
RECOC1_CD_01_dm01cel01 ONLINE
RECOC1_CD_02_dm01cel01 ONLINE
RECOC1_CD_03_dm01cel01 ONLINE
RECOC1_CD_04_dm01cel01 ONLINE
RECOC1_CD_05_dm01cel01 ONLINE
RECOC1_CD_06_dm01cel01 ONLINE
RECOC1_CD_07_dm01cel01 ONLINE
RECOC1_CD_08_dm01cel01 ONLINE
RECOC1_CD_09_dm01cel01 ONLINE
RECOC1_CD_10_dm01cel01 ONLINE
RECOC1_CD_11_dm01cel01 ONLINE

9. Recreate the flash cache

[root@dm01cel01 ~]# cellcli -e create flashcache all
Flash cache dm01cel01_FLASHCACHE successfully created

10. Check the status of the cell to confirm that it’s now in WriteBack mode:
[root@dm01cel01 ~]# cellcli -e list cell detail | grep flashCacheMode
flashCacheMode: writeback

11. Repeat these same steps again on the next cell. However, before taking another storage server offline, execute the following making sure ‘asmdeactivationoutcome’ displays YES:

[root@dm01cel01 ~]# cellcli -e list griddisk attributes name,asmmodestatus,asmdeactivationoutcome
DATAC1_CD_00_dm01cel01 ONLINE Yes
DATAC1_CD_01_dm01cel01 ONLINE Yes
DATAC1_CD_02_dm01cel01 ONLINE Yes
..
..
DBFS_DG_CD_02_dm01cel01 ONLINE Yes
DBFS_DG_CD_03_dm01cel01 ONLINE Yes
DBFS_DG_CD_04_dm01cel01 ONLINE Yes
..
..
RECOC1_CD_00_dm01cel01 ONLINE Yes
RECOC1_CD_01_dm01cel01 ONLINE Yes
..
..

Enable Write Back Flash Cache using a NON-ROLLING method
Enable write-back flash cache while RDBMS & ASM Instances are down.

1. Shutdown all databases and the entire cluster

Ensure the entire cluster is shut down (all databases and CRS). Not doing so will result in a crash of the all databases and the cluster when the cellsrv service is shut down and restarted. It is recommended to shut down each database with the srvctl command. However, you can shut down the entire cluster at once logged on as root on one of the compute nodes as follows:

[root@dm01db01 ~]# cd $GRID_HOME/bin
[root@dm01db01 ~]# ./crsctl stop cluster -all

2. Drop the flash cache on all cell
[root@dm01db01 ~]# dcli -g ~/cell_group -l root cellcli -e drop flashcache

3. Shut down cellsrv service
[root@dm01db01 ~]# dcli -g ~/cell_group -l root cellcli -e alter cell shutdown services cellsrv

4. Set the cell flashcache mode to writeback
[root@dm01db01 ~]# dcli -g ~/cell_group -l root cellcli -e “alter cell flashCacheMode=writeback”

5. Restart the cellsrv service
[root@dm01db01 ~]# dcli -g ~/cell_group -l root cellcli -e alter cell startup services cellsrv

6. Recreate the flash cache
[root@dm01db01 ~]# dcli -g ~/cell_group -l root cellcli -e create flashcache all

7. Check the status of all cells’ flash cache state
[root@dm01db01 ~]# dcli -g cell_group -l root cellcli -e list cell detail | grep flashCacheMode

The flashCacheMode state of all cells should now be in writeback

8. Start up the cluster and all databases. As root on one of the compute nodes, issue the following command:

[root@dm01db01 ~]# cd $GRID_HOME/bin
[root@dm01db01 ~]# ./crsctl start cluster -all

Enjoy!

Leave a Reply

Your email address will not be published.