Build Your Own Oracle Infrastructure: Part 0 – Master Index

Quick links to all the major sections in the series:

  1. Hardware.
  2. Software.
  3. Install Oracle VM Software.
  4. Configure Oracle VM Server.
  5. Create VMs.
  6. Build OEM Cloud Control Server.
  7. Build Oracle RAC Servers.
  8. SSH, DNS & CVU.
  9. Build Shared Storage Server.
  10. Install Oracle GI & Database Software.
  11. Oracle RAC Databases: Create, Test & Administer.
  12. Use Oracle Recovery Manager. (In Progress)
  13. Configure Oracle Data Guard. (Coming Soon)
  14. Install, Configure & Use Oracle GoldenGate. (Coming Soon)
Part Section or Task Notes
Part 1: Hardware.
Hardware Summary Approximate Costs
Oracle VM Server HP ProLiant DL380
Oracle Database 12c & Oracle VM Manager Server Customized PC
Shared Storage Server Customized PC
Gigabit Network Switch #1 D-Link 16 port
Gigabit Network Switch #2 D-Link 8 port
KVM 4 Port USB/PS2
Screen ViewSonic LCD
Keyboard PS/2
Mouse USB
UPS #1 Tripp Lite
UPS #2 CyberPower
Network Cables Cat 6
Shelving InterMetro
Cooling Fan 20" Box Fan
Part 2: Software.
Oracle VM Virtualization
Oracle Linux Operating System
Oracle VM Console Virtualization Console
Oracle Database 12c Database
Oracle Enterprise Manager Cloud Control Management Console
Oracle Enterprise Manager Repository Database Template Database Template
Oracle ASMlib rpm Storage RPM
Oracle GoldenGate Data Replication
Openfiler Shared Storage
FileZilla Secure FTP
Putty Command Line Interface
ImgBurn ISO Burner
X-Win32 X Windows Server
Part 3: Install Oracle VM Software.
Task #1: Install Oracle VM Server Installation steps for OVM Server
Task #2: Install Oracle VM Manager Installation steps for OVM Manager
Part 4: Configure OVM Server.
Task #1: Discover the OVM Server Start using OVM Manager
Task #2: Create a Server Pool Create the default Server Pool
Task #3: Create Storage Repositories Carve up the storage
Task #4: Create VM Networks Virtual Machines need Virtual Networks
Part 5: Create VMs.
Task #1: Create ORAEMCC_VM Cloud Control VM
Task #2: Create ORASVR01_VM DB Server VM
Task #3: Create RACNODE1_VM RAC node VM
Part 6: Build OEM Cloud Control Server.
Task #1: Install Oracle Linux OS installation
Task #2: Configure Oracle Linux OS configuration
Task #3: Install Oracle Database 12c DB software installation
Task #4: Create a Repository Database Use an Oracle template
Task #5: Install OEM Cloud Control OEM software installation
Task #6: Starting and Stopping Cloud Control Essential admin tasks
Part 7: Build Oracle RAC Servers.
Task #1: Install Oracle Linux on RACNODE1_VM OS installation
Task #2: Configure Oracle Linux on racnode1 OS configuration
Task #3: Clone RACNODE1_VM Shortcut to creating additional VMs
Task #4: Modify racnode2 Customize a clone of racnode1
Part 8: SSH, DNS & CVU.
Task #1: Setup SSH Secure Shell setup
Task #2: Configure DNS Domain Name System setup
Task #3: Run the CVU Cluster Verification Utility execution
Part 9: Build Shared Storage Server.
Task #1: Install Openfiler OS installation
Task #2: Configure Openfiler Shared disk configuration
Task #3: Configure iSCSI Targets in Oracle Linux Expose shared storage to the OS
Task #4: Configure Oracle ASM Automatic Storage Management setup
Part 10: Install Oracle GI & Database Software.
Task #1: Install the OEM Cloud Control Management Agent Agent installation
Task #2: Run the Grid Infrastructure Installer GI software installation
Task #3: Run the Database Installer DB software installation
Part 11: Oracle RAC Database: Create, Test & Administer.
Task #1: Creation Additional ASM Diskgroups Expand the storage footprint
Task #2: Create an Oracle 12c RAC Database Run the Db installer
Task #3: Test the Oracle 12c Grid Infrastructure and RAC Database Some test cases to try
Task #4: Common Administration Tasks Some admin you need to know
Part 12: Use Oracle Recovery Manager.
Task #1: Create an RMAN Catalog Create an RMAN schema in a DB
Task #2: Execute RMAN Backups Check out different backups
Task #3: Execute RMAN Restores & Recoveries Let's get the data back
Task #4: More Advanced Uses of RMAN Test some fun scenarios
Task #5: RMAN Reporting Find out what's going on

Oracle Enterprise Manager

Some tips, tricks and tasks to help you navigate the wonderful world of Oracle Enterprise Manager.

Quick links to all the tips, tricks and tasks:

Changing the SYSMAN Password.

In the good old days, you could just login to the repository database and use an ALTER USER SQL command to change the password of the SYSMAN user. In an object lesson of how to make things more complicated, here’s how you have to change the SYSMAN password now. There are two slightly different methods described below. One worked for OEM 12c R5 (you know the current SYSMAN password) and the other worked for OEM 13c R2 (you do not know the current SYSMAN password). These steps and others are described in more detail in MOS Doc ID 1365930.1:

OEM 12c R5 Method (you know the current SYSMAN password):

Locate the OMS_HOME directory:

[oracle@oraemcc ~]$ cd /u01/app/oracle/product/middleware/oms/bin

Shutdown the middle tier:

[oracle@oraemcc bin]$ ./emctl stop oms
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
Stopping WebTier...
WebTier Successfully Stopped
Stopping Oracle Management Server...
Oracle Management Server Successfully Stopped
Oracle Management Server is Down

Change the password:

[oracle@oraemcc bin]$ ./emctl config oms -change_repos_pwd
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
Enter Repository User's Current Password : <enter-old-SYSMAN-password>
Enter Repository User's New Password : <enter-new-SYSMAN-password>

Changing passwords in backend ...
Passwords changed in backend successfully.
Updating repository password in Credential Store...
Successfully updated Repository password in Credential Store.
Restart all the OMSs using 'emctl stop oms -all' and 'emctl start oms'.
Successfully changed repository password.

Stop the admin server process:

[oracle@oraemcc bin]$ ./emctl stop oms -all
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
Stopping WebTier...
WebTier Successfully Stopped
Stopping Oracle Management Server...
Oracle Management Server Already Stopped
AdminServer Successfully Stopped
Oracle Management Server is Down

Restart the middle tier:

[oracle@oraemcc bin]$ ./emctl start oms
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
Starting Oracle Management Server...

Starting WebTier...
WebTier Successfully Started
Oracle Management Server Successfully Started
Oracle Management Server is Up

OEM 13c R2 (you do NOT know the current SYSMAN password):

Stop the OMS:

[oracle@oraemcc bin]$ pwd
/u01/app/oracle/product/middleware/bin

[oracle@oraemcc bin]$ ./emctl stop oms -all -force
Oracle Enterprise Manager Cloud Control 13c Release 2  
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
Stopping Oracle Management Server…
WebTier Successfully Stopped
Oracle Management Server Successfully Stopped
Oracle Management Server is Down
JVMD Engine is Down
BI Publisher is disabled, to enable BI Publisher on this host, use the 'emctl config oms -enable_bip' command
Stopping BI Publisher Server…
BI Publisher Server Already Stopped
BI Publisher is disabled, to enable BI Publisher on this host, use the 'emctl config oms -enable_bip' command 
AdminServer Successfully Stopped
BI Publisher Server is Down
BI Publisher is disabled, to enable BI Publisher on this host, use the 'emctl config oms -enable_bip' command

Stop the local Management Agent (just in case):

[oracle@oraemcc bin]$ /u01/app/oracle/product/agent/agent_13.2.0.0.0/bin/emctl stop agent
Oracle Enterprise Manager Cloud Control 13c Release 2  
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
Stopping agent … stopped.

Ensure you can log into the SYSMAN_OPSS repository database account. Change the password using SQL*Plus if necessary:

SQL> connect sysman_opss/<your-SYSMAN_OPSS-pwd>@EMPDBREPOS
Connected.

SQL> show user
USER is "SYSMAN_OPSS"

Update the SYSMAN_OPSS password:

[oracle@oraemcc bin]$ ./emctl config oms -update_ds_pwd -ds_name sysman-opss-ds -ds_pwd <your-SYSMAN_OPSS-pwd>
Oracle Enterprise Manager Cloud Control 13c Release 2  
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
Successfully updated the datasource

Start the AdminServer and change the SYSMAN password:

[oracle@oraemcc bin]$ ./emctl start oms -admin_only
Oracle Enterprise Manager Cloud Control 13c Release 2  
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
Starting Admin Server only…
Admin Server Successfully Started

[oracle@oraemcc bin]$ ./emctl config oms -change_repos_pwd -use_sys_pwd
Oracle Enterprise Manager Cloud Control 13c Release 2  
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
Enter SYS Password : <your-repository-database-SYS-pwd>
Enter Repository User's New Password : <your-new-SYSMAN-pwd>
Changing passwords in backend … 
Passwords changed in backend successfully.
Updating repository password in Credential Store…
Successfully updated Repository password in Credential Store.
Restart all the OMSs using 'emctl stop oms -all' and 'emctl start oms'.
Successfully changed repository password.

Re-start the OMS and local Management Agent:

[oracle@oraemcc bin]$ ./emctl start oms
Oracle Enterprise Manager Cloud Control 13c Release 2  
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
Starting Oracle Management Server…
WebTier Successfully Started
Oracle Management Server Successfully Started
Oracle Management Server is Up
JVMD Engine is Up

[oracle@oraemcc bin]$ ./emctl status oms
Oracle Enterprise Manager Cloud Control 13c Release 2  
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
WebTier is Up
Oracle Management Server is Up
JVMD Engine is Up
BI Publisher Server is Down
BI Publisher is disabled, to enable BI Publisher on this host, use the 'emctl config oms -enable_bip' command

[oracle@oraemcc bin]$ /u01/app/oracle/product/agent/agent_13.2.0.0.0/bin/emctl /emctl start agent
Oracle Enterprise Manager Cloud Control 13c Release 2  
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
Starting agent ………………………… started.

Manually Remove Targets from the Repository.

From time to time you’ll need to remove targets and the Cloud Control GUI will complain, throw an error and invite you to contact Oracle Support. Like you, I have no time for that and fortunately there is a CLI method to achieve the desired outcome. Here’s a summary of the sub-programs within the MGMT_ADMIN package owned by SYSMAN (some are overloaded):

  • PROCEDURE ADD_TARGET_ADDITION_CALLBACK
  • PROCEDURE ADD_TARGET_DELETION_CALLBACK
  • PROCEDURE ADD_TARGET_DELETION_EXCEPTIONS
  • PROCEDURE CLEANUP_AGENT
  • PROCEDURE CLEAR_SITE_URL
  • PROCEDURE DELETE_OMS
  • PROCEDURE DELETE_TARGET
  • PROCEDURE DELETE_TARGET_ASYNC
  • PROCEDURE DELETE_TARGET_INTERNAL
  • PROCEDURE DELETE_TARGET_METRICS_1DAY
  • PROCEDURE DELETE_TARGET_METRICS_1HOUR
  • PROCEDURE DELETE_TARGET_METRICS_RAW
  • PROCEDURE DELETE_TARGET_SYNC
  • PROCEDURE DELETE_TARGET_WITH_MEMBERS
  • PROCEDURE DEL_TARGET_ADDITION_CALLBACK
  • PROCEDURE DEL_TARGET_DELETION_CALLBACK
  • PROCEDURE DEL_TARGET_DELETION_EXCEPTIONS
  • PROCEDURE DEREGISTER_TGT_DEL_MATCH
  • PROCEDURE DISABLE_METRIC_DELETION
  • PROCEDURE ENABLE_METRIC_DELETION
  • FUNCTION GET_MS_NAME RETURNS VARCHAR2
  • FUNCTION GET_OMS_DATA RETURNS REF CURSOR
  • FUNCTION GET_OMS_STATUS RETURNS NUMBER
  • FUNCTION GET_OMS_URLS RETURNS REF CURSOR
  • FUNCTION GET_SITE_URL RETURNS VARCHAR2
  • FUNCTION IS_METRIC_DELETION_ENABLED RETURNS NUMBER(38)
  • PROCEDURE REGISTER_TGT_DEL_MATCH
  • PROCEDURE SET_INACTIVE_TIME
  • PROCEDURE SET_LOG_LEVEL
  • PROCEDURE SET_LOG_PURGE
  • PROCEDURE SET_SITE_URL

In SQL*Plus, query the official name and type of the target you want to delete:

select target_name, 
       target_type                   
from   mgmt_targets 
where  target_name like 'orasvr02%';


TARGET_NAME                    TARGET_TYPE
------------------------------ --------------------
orasvr02.mynet.com             host
orasvr02.mynet.com:3872        oracle_emd

Delete the desired target:

exec mgmt_admin.delete_target('orasvr02.mynet.com','host');

To delete the Management Agent:

exec mgmt_admin.cleanup_agent('orasvr02.mynet.com:3872');

Oracle Database 19c

In this section, we’ll cover some database features and configurations using Oracle Database 19c Release 3 running on Oracle Linux 7.

Quick links to all the tasks:

Task #1: Installing Oracle Database 19c.

Nothing much has changed since Oracle Database 18c as far as installation of the binaries is concerned. You still have to create your own ORACLE_HOME directory, copy the downloaded zip file to that directory, unzip it then run the installer. Let’s crack on:

[oracle@orasvr01 dbhome_1]$ pwd
/u01/app/oracle/product/19.3.0/dbhome_1

[oracle@orasvr01 dbhome_1]$ ls -l *.zip
-rw-r--r-- 1 oracle oinstall 3059705302 Nov 13 09:34 LINUX.X64_193000_db_home.zip
[oracle@orasvr01 dbhome_1]$ unzip LINUX.X64_193000_db_home.zip

[oracle@orasvr01 dbhome_1]$ export ORACLE_BASE=/u01/app/oracle

[oracle@orasvr01 dbhome_1]$ export DISPLAY=<your-workstation-or-IP>:0.0

[oracle@orasvr01 dbhome_1]$ ./runInstaller
Select Set Up Software Only then click Next
Select Single instance database installation then click Next
Select Enterprise Edition then click Next
Check your ORACLE_BASE and ORACLE_HOME values are what you want then click Next
Your Linux group configuration should be picked up, click Next
We will run the root scripts interactively, click Next
Let the installer run its prerequisite checks
If you followed the configuration of Oracle Linux 7, you should see this screen, click Install
Let the installer do its thing until the root script window pops up
In a separate terminal session, run the root script then return to the pop up and click OK
[root@orasvr01 ~]# /u01/app/oracle/product/19.3.0/dbhome_1/root.sh
Performing root user operation.
The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/oracle/product/19.3.0/dbhome_1

Enter the full pathname of the local bin directory: [/usr/local/bin]: 
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Oracle Trace File Analyzer (TFA - Standalone Mode) is available at :
    /u01/app/oracle/product/19.3.0/dbhome_1/bin/tfactl

Note :
1. tfactl will use TFA Service if that service is running and user has been granted access
2. tfactl will configure TFA Standalone Mode only if user has no access to TFA Service or TFA is not installed 
Et voila! The installation is that simple!

Oracle Database 11g

Some notes about various 11g-isms which will hopefully keep you out of trouble.

Installing Oracle Database 11g Release 11.2.0.4 on Oracle Linux 7.

You’d think installing 11.2.0.4, which we’ve all done a thousand times, would be a stroll in the park, right? Wrong! On Oracle Linux 7 you will run into some gotchas. Fortunately, the following will speed you on your way to a successful installation.

The installer pre-req checks fail for a number of missing packages:

Run these yum commands to install what you need:

[root@orasvr01 ~]# yum install gcc
[root@orasvr01 ~]# yum install gcc-c++
[root@orasvr01 ~]# yum install elfutils-libelf-devel

This still leaves one package which fails the pre-req check, pdksh-5.2.14. This is caused by bug #19947777, but can safely be ignored. So either click the Ignore All checkbox and let the installer continue or run the installer again using this command (assuming you have fixed the other pre-req check failures):

[oracle@orasvr01 database]$ ./runInstaller –ignorePrereq

At around 86% complete, the installer throw another error. You’ll see this pop up:

Click Continue and you’ll be fine

The installer log file should contain this text at the of the file:

/usr/bin/ld: warning: -z nolazyload ignored.
/u01/app/oracle/product/11.2.0/dbhome_1/sysman/lib//libnmectl.a(nmectlt.o): In function nmectlt_genSudoProps': nmectlt.c:(.text+0x76): undefined reference toB_DestroyKeyObject'
nmectlt.c:(.text+0x7f): undefined reference to B_DestroyKeyObject' nmectlt.c:(.text+0x88): undefined reference toB_DestroyKeyObject'
nmectlt.c:(.text+0x91): undefined reference to `B_DestroyKeyObject'
collect2: error: ld returned 1 exit stat
INFO: us
make[1]: *** [/u01/app/oracle/product/11.2.0/dbhome_1/sysman/lib/emdctl] Error 1
INFO: make[1]: Leaving directory `/u01/app/oracle/product/11.2.0/dbhome_1/sysman/lib'
INFO: make: *** [emdctl] Error 2
INFO: End output from spawned process.
INFO: ----------------------------------
INFO: Exception thrown from action: make
Exception Name: MakefileException
Exception String: Error in invoking target 'agent nmhs' of makefile '/u01/app/oracle/product/11.2.0/dbhome_1/sysman/lib/ins_emagent.mk'.

This is caused by bug #19692824. You can download and install the patch or safely ignore the error as per MOS Doc ID 1965691.1.

Oracle Database 18c

Starting with Oracle Database 18c, Oracle have made a few changes. The biggest one perhaps is the intention to make quarterly releases each year from now on. Hence, Oracle Database 18c Release 1 in Q1 of 2018 (18.1), Release 2 in Q2 (18.2) and so on. Not all of these releases will be or are available for download for on premise installation. At the time of writing, 18.3 is only available for download for 4 different platforms and 18.4 is only available for a further 3 platforms. We’ll be using 18.3 for Linux x86-64 for a series of Oracle Database 18c articles. Enjoy!

Quick links to all the tasks:

Task #1: Deinstalling Oracle Database 18c.

Also starting with Oracle Database 18c, Oracle have changed the way the database software is installed. Previously, you’d grab the zip file(s) from OTN or wherever, copy them to some staging area on your target server, unzip them then run the installer to actually install the code set into your OFA designated path (if you were doing things properly that is). That’s now changed.

Nowadays, you still grab the zip file(s) from Oracle, but now you have to copy them directly to your pre-created OFA designated path, unzip them and run the installer to install the code set in that directory. In other words, the installer will not create an OFA path for you like it has previously. I’m sure this will lead to many installations being in the wrong place, so let’s start off my running a deinstall so you can get it right the second time.

I’ve installed Oracle Database 18c Release 3 (18.3) here:

[oracle@orasvr01 dbhome_1]$ pwd
/u01/app/oracle/product/18.0.0/dbhome_1

Given it’s 18.3, following OFA rules the path should be /u01/app/oracle/product/18.3.0/dbhome_1. So let’s uninstall it. These steps assume you haven’t created a database already. If you have, the deinstall process will detect it (and any associated Listener) and remove those too:

[oracle@orasvr01 dbhome_1]$ cd deinstall
[oracle@orasvr01 deinstall]$ pwd
/u01/app/oracle/product/18.0.0/dbhome_1/deinstall

[oracle@orasvr01 deinstall]$ ./deinstall
Checking for required files and bootstrapping …
Please wait …
Location of logs /u01/app/oraInventory/logs/
############ ORACLE DECONFIG TOOL START ############

################### DECONFIG CHECK OPERATION START ###################
## [START] Install check configuration ##

Checking for existence of the Oracle home location /u01/app/oracle/produc/18.0.0/dbhome_1
Oracle Home type selected for deinstall is: Oracle Single Instance Database
Oracle Base selected for deinstall is: /u01/app/oracle
Checking for existence of central inventory location /u01/app/oraInventory

## [END] Install check configuration

Network Configuration check config START

Network de-configuration trace file location: /u01/app/oraInventory/logs/netdc_check2019-07-12_11-27-40AM.log

Network Configuration check config END

Database Check Configuration START

Database de-configuration trace file location: /u01/app/oraInventory/logs/databasedc_check2019-07-12_11-27-40AM.log

Use comma as separator when specifying list of values as input

Specify the list of database names that are configured in this Oracle home []: 

Database Check Configuration END

################### DECONFIG CHECK OPERATION END  ###################  

################# DECONFIG CHECK OPERATION SUMMARY  ################# 
Oracle Home selected for deinstall is: /u01/app/oracle/product/18.0.0/dbhome_1
Inventory Location where the Oracle home registered is: /u01/app/oraInventory
Do you want to continue (y - yes, n - no)? [n]: y
A log of this session will be written to: '/u01/app/oraInventory/logs/deinstall_deconfig2019-07-12_11-27-39-AM.out'
Any error messages from this session will be written to: '/u01/app/oraInventory/logs/deinstall_deconfig2019-07-12_11-27-39-AM.err'

################## DECONFIG CLEAN OPERATION START  ##################  
Database de-configuration trace file location: /u01/app/oraInventory/logs/databasedc_clean2019-07-12_11-27-40AM.log

Network Configuration clean config START

Network de-configuration trace file location: /u01/app/oraInventory/logs/netdc_clean2019-07-12_11-27-40AM.log

De-configuring Naming Methods configuration file…
Naming Methods configuration file de-configured successfully.

De-configuring Local Net Service Names configuration file…
Local Net Service Names configuration file de-configured successfully.

De-configuring backup files…
Backup files de-configured successfully.

The network configuration has been cleaned up successfully.

Network Configuration clean config END

################### DECONFIG CLEAN OPERATION END ###################  

################# DECONFIG CLEAN OPERATION SUMMARY #################  
#############################################################

####### ORACLE DECONFIG TOOL END #######
  
Using properties file /tmp/deinstall2019-07-12_11-27-30AM/response/deinstall_2019-07-12_11-27-39-AM.rsp
Location of logs /u01/app/oraInventory/logs/

###### ORACLE DEINSTALL TOOL START ######  

################# DEINSTALL CHECK OPERATION SUMMARY #################  
A log of this session will be written to: '/u01/app/oraInventory/logs/deinstall_deconfig2019-07-12_11-27-39-AM.out'
Any error messages from this session will be written to: '/u01/app/oraInventory/logs/deinstall_deconfig2019-07-12_11-27-39-AM.err'

################## DEINSTALL CLEAN OPERATION START ##################  
## [START] Preparing for Deinstall ##
Setting LOCAL_NODE to orasvr01
Setting CRS_HOME to false
Setting oracle.installer.invPtrLoc to /tmp/deinstall2019-07-12_11-27-30AM/oraInst.loc
Setting oracle.installer.local to false
## [END] Preparing for Deinstall ##

Setting the force flag to false
Setting the force flag to cleanup the Oracle Base
Oracle Universal Installer clean START

Detach Oracle home '/u01/app/oracle/product/18.0.0/dbhome_1' from the central inventory on the local node : Done

Delete directory '/u01/app/oracle/product/18.0.0/dbhome_1' on the local node : Done

The Oracle Base directory '/u01/app/oracle' will not be removed on local node. The directory is in use by Oracle Home '/u01/app/oracle/product/19.3.0/dbhome_1'.

Oracle Universal Installer cleanup was successful.

Oracle Universal Installer clean END

## [START] Oracle install clean ##

## [END] Oracle install clean ##

################### DEINSTALL CLEAN OPERATION END ###################  

################# DEINSTALL CLEAN OPERATION SUMMARY #################  
Successfully detached Oracle home '/u01/app/oracle/product/18.0.0/dbhome_1' from the central inventory on the local node.
Successfully deleted directory '/u01/app/oracle/product/18.0.0/dbhome_1' on the local node.
Oracle Universal Installer cleanup was successful.

Review the permissions and contents of '/u01/app/oracle' on nodes(s) 'orasvr01'.
If there are no Oracle home(s) associated with '/u01/app/oracle', manually delete '/u01/app/oracle' and its contents.
Oracle deinstall tool successfully cleaned up temporary directories.
#######################################################################

####### ORACLE DEINSTALL TOOL END #######  

Task #2: Installing Oracle Database 18c (GUI).

There’s really not much to write home about with regards to installing 18c compared to previous versions. It has changed just a little so let’s quickly run through what the GUI installation looks like:

[oracle@orasvr01 dbhome_1]$ pwd
/u01/app/oracle/product/18.3.0/dbhome_1

[oracle@orasvr01 dbhome_1]$ ls -l
-rw-r--r-- 1 oracle oinstall 4564649047 Jul 12 11:40 LINUX.X64_180000_db_home.zip

[oracle@orasvr01 dbhome_1]$ unzip LINUX.X64_180000_db_home.zip
(output is boring so I didn't include it)

[oracle@orasvr01 dbhome_1]$ export ORACLE_BASE=/u01/app/oracle
[oracle@orasvr01 dbhome_1]$ export DISPLAY=<your_workstation_or_IP>:0.0
[oracle@orasvr01 dbhome_1]$ ./runInstaller
Select Set Up Software Only then click Next
Select Single instance database installation then click Next
Select Enterprise Edition then click Next
This screen tells you where the installer will install the software. Click Next.
These groups are created beforehand here. Click Next.
The installer runs the pre-req checks. It only takes a few seconds.
Correct any pre-req check failures then click Next.

In my case, I haven’t configured enough swap space. Since this is a non-production system this doesn’t matter so I can safely check the Ignore All check box, click Next and move on. Incidentally, if you need to configure more swap via a swap file, check this out.

You see this only if you chose to ignore some pre-req check failures.
Review your settings then click Install.
Tick tock tick tock…
If all goes well, the installer tells you to run the root.sh script.
[root@orasvr01 ~]# /u01/app/oracle/product/18.3.0/dbhome_1/root.sh
Performing root user operation.

The following environment variables are set as:
     ORACLE_OWNER= oracle
     ORACLE_HOME=  /u01/app/oracle/product/18.3.0/dbhome_1

Enter the full pathname of the local bin directory: [/usr/local/bin]: 
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Do you want to setup Oracle Trace File Analyzer (TFA) now ? yes|[no] : no
Oracle Trace File Analyzer (TFA - Non Daemon Mode) is available at :
     /u01/app/oracle/product/18.3.0/dbhome_1/suptools/tfa/release/tfa_home/bin/tfactl

Note :
1. tfactl will use TFA Daemon Mode if TFA already running in Daemon Mode and user has access to TFA
2. tfactl will configure TFA Non Daemon Mode only if user has no access to TFA Daemon mode or TFA Daemon mode is not installed 

OR

Oracle Trace File Analyzer (TFA - Daemon Mode) can be installed by running this script :
     /u01/app/oracle/product/18.3.0/dbhome_1/suptools/tfa/release/tfa_home/install/roottfa.sh

Click OK in the Execute Configuration Scripts window.

Click Close and you’re done.

Task #3: Installing Oracle Database 18c (silent install).

Although it doesn’t take that long to install the database software, sometimes you just haven’t got the time (or patience) to sit there and click-click-click. Silent (non-interactive) installations have been around for a while and Oracle Database 18c also supports that method. Rather useful if you’ve got a load of installations to do and need to (mostly) automate them.

Silent installations reply upon a response file which contains most or all of the values you would be prompted for if you were to run an interactive installation. Response files have a specific format which you must adhere to. A database software installation response file template can be found in $ORACLE_HOME/install/response/db_install.rsp. A more effective way to generate a populated response file is to run the GUI installation and save the response file towards the end of that process. That’s what I did to demonstrate the silent installation coming up shortly.

I always use a staging area to store the database media zip file. To speed up the copying, unzipping and installation of the software, I use this simple script. The significant line of code which runs the silent installation is this:

${ORACLE_HOME}/runInstaller -silent -responseFile ${RESPFILE} 

Here is the screen output from running the script as the oracle user:

[oracle@orasvr01 shell]$ ./silent_db_install.sh -help

Usage: silent_db_install.sh DB_Release Zip_File_Path Response_File_Path DELETE|NODELETE

       where:     
       DB_Release               = Database_Version.Database_Release.0 (e.g. 18.3.0)     
       Zip_File_Path            = Full path to the database binaries zip file     
       Response_File_Path       = Full path to the silent install response file     
       DELETE|NODELETE          = Delete the zip file in the new ORACLE_HOME (DELETE) or not (NODELETE)

[oracle@orasvr01 shell]$ ./silent_db_install.sh 18.3.0 /u01/MEDIA/database/18c/LINUX.X64_180000_db_home.zip /home/oracle/scripts/shell/install_183.rsp DELETE

     *** Checking parameters     
     *** Copying zip file: /u01/MEDIA/database/18c/LINUX.X64_180000_db_home.zip to the new 
              ORACLE_HOME: /u01/app/oracle/product/18.3.0/dbhome_1     
     *** Unzipping LINUX.X64_180000_db_home.zip into /u01/app/oracle/product/18.3.0/dbhome_1     
     *** Deleting /u01/app/oracle/product/18.3.0/dbhome_1/LINUX.X64_180000_db_home.zip     
     *** Passing execution to the Oracle Installer in silent mode
 
Launching Oracle Database Setup Wizard…
The response file for this session can be found at:
 /u01/app/oracle/product/18.3.0/dbhome_1/install/response/db_2019-08-16_12-10-14PM.rsp
 
You can find the log of this install session at:
 /tmp/InstallActions2019-08-16_12-10-14PM/installActions2019-08-16_12-10-14PM.log
 
As a root user, execute the following script(s):
         1. /u01/app/oraInventory/orainstRoot.sh
         2. /u01/app/oracle/product/18.3.0/dbhome_1/root.sh
 
Execute /u01/app/oraInventory/orainstRoot.sh on the following nodes: 
 [orasvr01]

Execute /u01/app/oracle/product/18.3.0/dbhome_1/root.sh on the following nodes: 
 [orasvr01]

Successfully Setup Software.
Moved the install session logs to:
 /u01/app/oraInventory/logs/InstallActions2019-08-16_12-10-14PM

     *** Please check the Oracle Installer output for additional root scripts to run.

Note, the first database software installation does take into account the absence of an Oracle inventory. Subsequent installations on the same server leverage a preexisting inventory. Also note, I had to fix the swap space issue otherwise the silent installation complained. Fixing a low swap space issue is documented here.

For completeness, here’s the output from the two root scripts referenced in the above output:

[root@orasvr01 ~]# /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.
Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.

[root@orasvr01 app]# /u01/app/oracle/product/18.3.0/dbhome_1/root.sh
Check /u01/app/oracle/product/18.3.0/dbhome_1/install/root_orasvr01.mynet.com_2019-08-16_12-12-24-830643173.log for the output of root script

The script can be run multiple times on the same server. Each time it will create a new OFA compliant ORACLE_HOME, copy the zip file to it, unzip it, run a silent installation and optionally delete the zip file from the new ORACLE_HOME (but not from the staging area). Hopefully it will save you some time.

WordPress

Is it just me or is WordPress a pain?

They keep changing things and (almost) every time they do it messes up my website. The spacing and layout gets all screwed up and it takes me days to fix it. I thought WordPress and WordPress themes were supposed to make this stuff quick and easy. Erm….nope.

So if you’re reading this and wondering why some of my pages look like my cat designed them, I’m currently fixing pretty much everything. WordPress has embraced a block layout because apparently the now “classic” editor wasn’t good enough. Therein lies the problem. Bear with me.

Oracle Linux 7 Installation & Configuration

Introduction.

Getting ready to install, configure and use Oracle Database 18c and 19c meant rebuilding my two stand alone database servers with Oracle Linux 7. In time honored tradition of changing things for no good reason, setting up Oracle Linux 7 is slightly different to version 6 in a number of important ways. Not annoying at all then. So here we’ll run through installing and configuring Oracle Linux 7 Update 6 (OL7.6).

The two servers we’ll be rebuilding are called orasvr01 and orasvr02. The original build of orasvr01 can be found here. The rebuild will be similar, but this time around we’ll use Openfiler for the database storage. The orasvr01 server will use regular file system storage and orasvr02 will use ASM. The root and /u01 file systems will be allocated from the VM_Filesystems_Repo storage repository, just as they were before.

For the most part, installing and configuring OL7.6 will be the same for both orasvr01 and orasvr02, so we’ll mainly focus on orasvr01. The differences will come when setting up the Openfiler disks for file systems (orasvr01) and ASM (orasvr02). I’ll explain those differences in the relevant sections below. Let’s get started!

Quick links to all the tasks:

Task #1: Create the VM.

In OVM Manager, click the Create Virtual Machine icon. Ensure the “Create a new VM (Click ‘Next’ to continue)” radio button is selected, then click Next. Use these values to populate the next screen:

Note, I recently doubled the memory of the Oracle VM server from 72 GB to 144 GB. Hence, I was able to increase the memory of the orasvr01/02 VMs from 8192 MB to 16,384 MB.

Click Next. Add the Management_Public and Shared_Storage_Public networks so your screen looks like this:

Click Next. Add two virtual disks for the operating system and /u01 file system. Then add a CD/DVD containing the ISO for Oracle Linux 7 Update 6. Your screen should look like this:

Click Next. Now change the boot order so when you start the VM for the first time, it will boot from the Linux ISO in the virtual CD/DVD drive. Your screen should look like this:

Click Finish and you’re done.

Task #2: Install Oracle Linux.

In OVM Manager start ORASVR01_VM, wait for a few seconds then connect to the console. You’ll see this opening screen:

Once the installation kicks off, the CD/DVD drive will be mounted and the media checked:

After a few moments, the Welcome screen will appear where you will choose your language:

I had the separated double mouse pointer issue again, but was able to ‘fix’ it by moving the mouse pointer to a corner of the screen and getting the two pointers to superimpose. Once they are, don’t move the pointer too quickly or they’ll separate again.

Choose your language then click Continue. The Installation Summary screen appears next. Use the values below or choose your own:

Category Option Value Comments
LOCALIZATION
DATE & TIME Americas/Chicago timezone Your choice
LANGUAGE SUPPORT English (United States) Your choice
KEYBOARD English (US) Your choice
SOFTWARE
INSTALLATION SOURCE Local Media OL 7.6 ISO in the CD/DVD drive
SOFTWARE SELECTION Infrastructure Server Plus Add-Ons: System Administration Tools
SYSTEM
INSTALLATION DESTINATION Automatic partitioning selected Use the 40 GiB xvda to install the OS
NETWORK & HOSTNAME Wired (eth0) connected Only configure eth0 (public) and set the hostname
KDUMP Kdump is disabled Uncheck Enable kdump
SECURITY POLICY No profile selected Set Apply security policy to OFF

 

When you have configured each option, your screen should look like this:

Click Begin Installation. This will display the CONFIGURATION USER SETTINGS screen:

Click the ROOT PASSWORD icon. This will display a new screen allowing you to enter a password for the root user:

Enter your new root password, then click Done. This will return you to the CONFIGURATION USER SETTINGS screen where you can monitor the progress of the Oracle Linux package installation:

Once all the packages have installed you’ll see this screen:

Before you click the Reboot button, we need to eject the Oracle Linux ISO from the virtual CD/DVD drive. Otherwise rebooting will start the installation process again. In OVM Manager, edit the ORASVR01_VM virtual machine and click the Disks tab:

Click the Eject icon to remove the ISO file from the CD/DVD drive. Your screen will look like this:

Return to the VM console and click Reboot. This is where life may get a little interesting. I tried this process multiple times. Sometimes clicking the Reboot button worked. Sometimes, but not often. Other times the reboot just hung, so I had to stop and restart the VM using OVM Manager. Sometimes even that didn’t work and I had to resort to killing the VM in OVM Manager, then re-starting it. Another time I had to ‘destroy’ the VM and re-create it using xm commands on the OVM server. It sounds worse than it is. It’s a quick and simple procedure documented here.

Eventually, the reboot happens and you’ll see a Linux login prompt:

Don’t get too excited. There are a few configuration changes we need to make before OL7.6 is ready for prime time. Some of these could have been configured via the INSTALLATION SUMMARY screen, but I wanted to explicitly cover how some Linux administration has changed in version 7. Clicking around in the installation GUI won’t show you that. So without further ado, fire up Putty and login as root.

Task #3: Run an Update.

Oracle claim the yum repository is all ready to go in Oracle Linux 7. Well, sort of. Run a yum update:

[root@orasvr01 ~]# yum update 

There will probably be plenty of things to update which is fine. Look away though and you might miss this message:

IMPORTANT: A legacy Oracle Linux yum server repo file was found. Oracle Linux yum server repository 
configurations have changed which means public-yum-ol7.repo will no longer be updated. New repository 
configuration files have been installed but are disabled. To complete the transition, run this script 
as the root user:
  
/usr/bin/ol_yum_configure.sh 

Fair enough, let’s run that and see what it does:

[root@orasvr01 ~]# /usr/bin/ol_yum_configure.sh
Repository ol7_UEKR5 already enabled
Repository ol7_latest already enabled 

Looks like we’re all set.

Task #4: Install X Windows.

Putty has its place, but I prefer working with a windows interface meant for adults. The command to install xterm can be found here. I use X-Win32 as my X Windows PC based server. It complained about not seeing xauth on the server side, so I installed that as well using this command:

[root@orasvr01 ~]# yum install xauth

Task #5: Disable SE Linux.

The simple edit to disable SE Linux can be found here.

Task #6: Turn Off Linux Firewall.

Managing the firewall has changed in Oracle Linux 7. By default, the firewall is provided via a daemon (firewalld) and is controlled by the systemctl command. Go here for the steps to disable the firewall in Oracle Linux 7.

Task #7: Configure Storage Networking (eth1).

The orasvr01 and orasvr02 servers have 2 NICs each. We’ve already configured the public interface (eth0). Now it’s time to configure the NIC which will connect the server to the storage coming from Openfiler (eth1).

Before we do that, replace the /etc/hosts file with our standard one which lists all our infrastructure IP addresses. That file can be found here.

Each NIC has a configuration file located in /etc/sysconfig/network-scripts. The file name follows the pattern ifcfg-ethN, where N is the number of the NIC you’re interested in. In this case, the file we want to edit is ifcfg-eth1. This is what OVM/Oracle Linux installer gave us by default:

[root@orasvr01 network-scripts]# cat ifcfg-eth1
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=dhcp
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=eth1
UUID=4eb42d7b-adcf-4389-9e6e-a006b3011424
DEVICE=eth1
ONBOOT=no

Edit ifcfg-eth1 so it looks like this:

TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=eth1
UUID=4eb42d7b-adcf-4389-9e6e-a006b3011424
DEVICE=eth1
ONBOOT=yes
IPADDR=200.200.20.17
PREFIX=24
GATEWAY=200.200.10.1
DNS1=200.200.10.1

Next, check the NIC configuration for eth1. As you can see, not much going on:

[root@orasvr01 network-scripts]# ifconfig -a eth1
eth1: flags=4163  mtu 1500
        ether 00:21:f6:ef:3f:da  txqueuelen 1000  (Ethernet)
        RX packets 311  bytes 14326 (13.9 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
 

Next, check the status of the network. Note eth0 gets a mention, but not eth1:

[root@orasvr01 network-scripts]# systemctl status network
● network.service - LSB: Bring up/down networking
   Loaded: loaded (/etc/rc.d/init.d/network; bad; vendor preset: disabled)
   Active: active (exited) since Mon 2019-07-01 12:06:10 CDT; 2h 15min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 1057 ExecStart=/etc/rc.d/init.d/network start (code=exited, status=0/SUCCESS)
Jul 01 12:06:09 orasvr01.mynet.com systemd[1]: Starting LSB: Bring up/down networking…
Jul 01 12:06:10 orasvr01.mynet.com network[1057]: Bringing up loopback interface:  [  OK  ]
Jul 01 12:06:10 orasvr01.mynet.com network[1057]: Bringing up interface eth0:  [  OK  ]
Jul 01 12:06:10 orasvr01.mynet.com systemd[1]: Started LSB: Bring up/down networking.

Start the eth1 interface:

[root@orasvr01 network-scripts]# ifup eth1

Now check the NIC configuration again:

[root@orasvr01 network-scripts]# ifconfig -a eth1
eth1: flags=4163  mtu 1500
        inet 200.200.20.17  netmask 255.255.255.0  broadcast 200.200.20.255
        inet6 fe80::908:e76e:8411:9051  prefixlen 64  scopeid 0x20
        ether 00:21:f6:ef:3f:da  txqueuelen 1000  (Ethernet)
        RX packets 371  bytes 17086 (16.6 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 63  bytes 9898 (9.6 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Signs of life! The eth1 interface is now up. Re-check the status of the network:

[root@orasvr01 network-scripts]# systemctl status network
● network.service - LSB: Bring up/down networking    
    Loaded: loaded (/etc/rc.d/init.d/network; bad; vendor preset: disabled)
    Active: active (exited) since Mon 2019-07-01 12:06:10 CDT; 2h 42min ago
      Docs: man:systemd-sysv-generator(8)
   Process: 1057 ExecStart=/etc/rc.d/init.d/network start (code=exited, status=0/SUCCESS)
Jul 01 12:06:09 orasvr01.mynet.com systemd[1]: Starting LSB: Bring up/down networking…
Jul 01 12:06:10 orasvr01.mynet.com network[1057]: Bringing up loopback interface:  [  OK  ]
Jul 01 12:06:10 orasvr01.mynet.com network[1057]: Bringing up interface eth0:  [  OK  ]
Jul 01 12:06:10 orasvr01.mynet.com systemd[1]: Started LSB: Bring up/down networking.

Note the output still only references the eth0 NIC. Re-start the network then re-check its status:

[root@orasvr01 network-scripts]# systemctl restart network

[root@orasvr01 network-scripts]# systemctl status network
 ● network.service - LSB: Bring up/down networking
    Loaded: loaded (/etc/rc.d/init.d/network; bad; vendor preset: disabled)
    Active: active (exited) since Mon 2019-07-01 14:51:59 CDT; 3s ago
      Docs: man:systemd-sysv-generator(8)
   Process: 1884 ExecStop=/etc/rc.d/init.d/network stop (code=exited, status=0/SUCCESS)
   Process: 2092 ExecStart=/etc/rc.d/init.d/network start (code=exited, status=0/SUCCESS)
 Jul 01 14:51:58 orasvr01.mynet.com systemd[1]: Starting LSB: Bring up/down networking…
 Jul 01 14:51:59 orasvr01.mynet.com network[2092]: Bringing up loopback interface:  [  OK  ]
 Jul 01 14:51:59 orasvr01.mynet.com network[2092]: Bringing up interface eth0:  Connection successfully activated (D-Bus ac…ion/7)
 Jul 01 14:51:59 orasvr01.mynet.com network[2092]: [  OK  ]
 Jul 01 14:51:59 orasvr01.mynet.com network[2092]: Bringing up interface eth1:  Connection successfully activated (D-Bus ac…ion/8)
 Jul 01 14:51:59 orasvr01.mynet.com network[2092]: [  OK  ]
 Jul 01 14:51:59 orasvr01.mynet.com systemd[1]: Started LSB: Bring up/down networking.
 Hint: Some lines were ellipsized, use -l to show in full.

The output now references eth1 and all seems well. We should now be able to ping the openfiler-storage IP address using the eth1 interface:

[root@orasvr01 network-scripts]# ping -I eth1 openfiler-storage
PING openfiler-storage (200.200.20.6) from 200.200.20.17 eth1: 56(84) bytes of data.
64 bytes from openfiler-storage (200.200.20.6): icmp_seq=1 ttl=64 time=0.393 ms
64 bytes from openfiler-storage (200.200.20.6): icmp_seq=2 ttl=64 time=0.174 ms
64 bytes from openfiler-storage (200.200.20.6): icmp_seq=3 ttl=64 time=0.175 ms

Yes! Get in! Storage networking is sorted. Onto the next task.

Task #8: Add Users & Groups.

The easiest way to setup the users and groups necessary to run Oracle Database instances on your server is to use Oracle’s preinstallation package. Since we’re going to use Oracle Database 12c Release 2, 18c and 19c we may as well go for the highest version available. That’ll be the one for Oracle Database 19c then:

[root@orasvr01 ~]# yum install oracle-database-preinstall-19c

Amongst other things, this package creates the oracle user and a bunch of groups using a default UID and default GIDs.

In /etc/passwd:

oracle:x:54321:54321::/home/oracle:/bin/bash

In /etc/group:

oinstall:x:54321:oracle
dba:x:54322:oracle
oper:x:54323:oracle
backupdba:x:54324:oracle
dgdba:x:54325:oracle
kmdba:x:54326:oracle
racdba:x:54330:oracle

It does not create a grid user or the various ASM groups you’ll need to install Grid Infrastructure. To add those and to fix the default IDs, I used this script (modify for your own needs). The script makes the necessary changes and returns this result:

oracle user id:  uid=1000(oracle) gid=1000(oinstall) groups=1000(oinstall),1007(asmdba),1001(dba),1002(oper),1003(backupdba),1004(dgdba),1005(kmdba),1006(racdba)

grid user id:  uid=1001(grid) gid=1000(oinstall) groups=1000(oinstall),1007(asmdba),1008(asmadmin),1009(asmoper),1001(dba),1006(racdba)

Task #9: Modify Shell & Resource Limits.

For both the oracle and grid users the value of umask must be any one of these 22, 022, 0022.

[oracle@orasvr01 ~]$ umask
0022

If it’s not the correct value, set it explicitly in the ~/.bash_profile file:

umask 022

User resource limits are usually defined in /etc/security/limits.conf. When using the Oracle pre-installation package, these limits are created for the oracle user in this file instead:

/etc/security/limits.d/oracle-database-preinstall-19c.conf

These limits need to be replicated for the grid user, so add two sets of entries to /etc/security/limits.conf. Your values may be different depending upon your hardware configuration:

# resource limits for oracle user:
oracle   soft   nofile   1024
oracle   hard   nofile   65536
oracle   soft   nproc    16384
oracle   hard   nproc    16384
oracle   soft   stack    10240
oracle   hard   stack    32768
oracle   hard   memlock  134217728
oracle   soft   memlock  134217728

# resource limits for grid user:
grid   soft   nofile   1024
grid   hard   nofile   65536
grid   soft   nproc    16384
grid   hard   nproc    16384
grid   soft   stack    10240
grid   hard   stack    32768
grid   hard   memlock  134217728
grid   soft   memlock  134217728

You can check these are operational by using this simple script.

Finally, if it so pleases you and it does me, change the insane alias defaults for ls, vi and grep in the oracle and grid user’s ~/.bash_profile:

unalias ls
unalias vi
unalias grep

Task #10: Configure iSCSI Storage.

There are options you can choose during the installation of Oracle Linux which will install the necessary iscsi packages. However, this is how you do it manually.

First check if the iscsi packages are installed:

[root@orasvr01 ~]# rpm -qa | grep iscsi
[root@orasvr01 ~]#

Nope, so let’s install them:

[root@orasvr01 ~]# yum install iscsi-initiator-utils

[root@orasvr01 ~]# rpm -qa | grep iscsi
iscsi-initiator-utils-iscsiuio-6.2.0.874-10.0.9.el7.x86_64
iscsi-initiator-utils-6.2.0.874-10.0.9.el7.x86_64

Next, enable and start the iscsid daemon:

[root@orasvr01 ~]# systemctl enable iscsid
Created symlink from /etc/systemd/system/multi-user.target.wants/iscsid.service to /usr/lib/systemd/system/iscsid.service.

[root@orasvr01 ~]# systemctl start iscsid

[root@orasvr01 ~]# systemctl status iscsid
● iscsid.service - Open-iSCSI
    Loaded: loaded (/usr/lib/systemd/system/iscsid.service; enabled; vendor preset: disabled)
    Active: active (running) since Thu 2019-07-04 17:45:10 CDT; 7min ago
      Docs: man:iscsid(8)
            man:iscsiadm(8)
  Main PID: 23489 (iscsid)
    Status: "Ready to process requests"
    CGroup: /system.slice/iscsid.service
            └─23489 /sbin/iscsid -f -d2
Jul 04 17:45:10 orasvr01.mynet.com systemd[1]: Starting Open-iSCSI…
Jul 04 17:45:10 orasvr01.mynet.com iscsid[23489]: iscsid: InitiatorName=iqn.1988-12.com.oracle:274b38e4651d
Jul 04 17:45:10 orasvr01.mynet.com iscsid[23489]: iscsid: InitiatorAlias=orasvr01.mynet.com
Jul 04 17:45:10 orasvr01.mynet.com iscsid[23489]: iscsid: Max file limits 1024 4096
Jul 04 17:45:10 orasvr01.mynet.com systemd[1]: Started Open-iSCSI.

I already carved up the /dev/sdc disk device in Openfiler (Western Digital 300 GB VelociRaptor SATA 3 drive) into two sets of 10 volumes. A set will be allocated to each server. Here’s a summary of the volume allocation:

Host iSCSI Target Disk Device (GB) File System/ASM Disk
orasvr01 iqn.2006-01.com.openfiler:orasvr01vg-vol01 /dev/sda (20) /u02
iqn.2006-01.com.openfiler:orasvr01vg-vol02 /dev/sdb (20) /u03
iqn.2006-01.com.openfiler:orasvr01vg-vol03 /dev/sdc (20) /u04
iqn.2006-01.com.openfiler:orasvr01vg-vol04 /dev/sdd (20) /u05
iqn.2006-01.com.openfiler:orasvr01vg-vol05 /dev/sde (20) /u06
iqn.2006-01.com.openfiler:orasvr01vg-vol06 /dev/sdf (20) /u07
iqn.2006-01.com.openfiler:orasvr01vg-vol13 /dev/sdg (10) N/A
iqn.2006-01.com.openfiler:orasvr01vg-vol15 /dev/sdh (1) N/A
iqn.2006-01.com.openfiler:orasvr01vg-vol16 /dev/sdi (1) N/A
iqn.2006-01.com.openfiler:orasvr01vg-vol17 /dev/sdj (1) N/A
orasvr02 iqn.2006-01.com.openfiler:orasvr02vg-vol07 /dev/sdj (20) DATA_000
iqn.2006-01.com.openfiler:orasvr02vg-vol08 /dev/sdi (20) DATA_001
iqn.2006-01.com.openfiler:orasvr02vg-vol09 /dev/sdh (20) DATA_002
iqn.2006-01.com.openfiler:orasvr02vg-vol10 /dev/sdg (20) RECO_000
iqn.2006-01.com.openfiler:orasvr02vg-vol11 /dev/sdf (20) RECO_001
iqn.2006-01.com.openfiler:orasvr02vg-vol12 /dev/sde (20) RECO_002
iqn.2006-01.com.openfiler:orasvr02vg-vol14 /dev/sdd (10) REDO_000
iqn.2006-01.com.openfiler:orasvr02vg-vol18 /dev/sdc (1) N/A
iqn.2006-01.com.openfiler:orasvr02vg-vol19 /dev/sdb (1) N/A
iqn.2006-01.com.openfiler:orasvr02vg-vol20 /dev/sda (1) N/A

First, let’s discover the iSCSI targets allocated to orasvr01:

[root@orasvr01 ~]# iscsiadm -m discovery -t sendtargets -p openfiler-storage
200.200.20.6:3260,1 iqn.2006-01.com.openfiler:orasvr01vg-vol17
200.200.10.6:3260,1 iqn.2006-01.com.openfiler:orasvr01vg-vol17
200.200.20.6:3260,1 iqn.2006-01.com.openfiler:orasvr01vg-vol16
200.200.10.6:3260,1 iqn.2006-01.com.openfiler:orasvr01vg-vol16
200.200.20.6:3260,1 iqn.2006-01.com.openfiler:orasvr01vg-vol15
200.200.10.6:3260,1 iqn.2006-01.com.openfiler:orasvr01vg-vol15
200.200.20.6:3260,1 iqn.2006-01.com.openfiler:orasvr01vg-vol13
200.200.10.6:3260,1 iqn.2006-01.com.openfiler:orasvr01vg-vol13
200.200.20.6:3260,1 iqn.2006-01.com.openfiler:orasvr01vg-vol06
200.200.10.6:3260,1 iqn.2006-01.com.openfiler:orasvr01vg-vol06
200.200.20.6:3260,1 iqn.2006-01.com.openfiler:orasvr01vg-vol05
200.200.10.6:3260,1 iqn.2006-01.com.openfiler:orasvr01vg-vol05
200.200.20.6:3260,1 iqn.2006-01.com.openfiler:orasvr01vg-vol04
200.200.10.6:3260,1 iqn.2006-01.com.openfiler:orasvr01vg-vol04
200.200.20.6:3260,1 iqn.2006-01.com.openfiler:orasvr01vg-vol03
200.200.10.6:3260,1 iqn.2006-01.com.openfiler:orasvr01vg-vol03
200.200.20.6:3260,1 iqn.2006-01.com.openfiler:orasvr01vg-vol02
200.200.10.6:3260,1 iqn.2006-01.com.openfiler:orasvr01vg-vol02
200.200.20.6:3260,1 iqn.2006-01.com.openfiler:orasvr01vg-vol01
200.200.10.6:3260,1 iqn.2006-01.com.openfiler:orasvr01vg-vol01

As before, orasvr01 sees the iSCSI targets on both the public network (200.200.10.x) and the storage network (200.200.20.x). I have no idea why, but needless to say we’re only interested in the targets on the storage network.

Next, we need to log into each iSCSI target:

[root@orasvr01 ~]# iscsiadm -m node -T iqn.2006-01.com.openfiler:orasvr01vg-vol01 -p 200.200.20.6 -l
Logging in to iface: default, target: iqn.2006-01.com.openfiler:orasvr01vg-vol01, portal: 200.200.20.6,3260
Login to [iface: default, target: iqn.2006-01.com.openfiler:orasvr01vg-vol01, portal: 200.200.20.6,3260] successful.

[root@orasvr01 ~]# iscsiadm -m node -T iqn.2006-01.com.openfiler:orasvr01vg-vol02 -p 200.200.20.6 -l
Logging in to iface: default, target: iqn.2006-01.com.openfiler:orasvr01vg-vol02, portal: 200.200.20.6,3260
Login to [iface: default, target: iqn.2006-01.com.openfiler:orasvr01vg-vol02, portal: 200.200.20.6,3260] successful.

[root@orasvr01 ~]# iscsiadm -m node -T iqn.2006-01.com.openfiler:orasvr01vg-vol03 -p 200.200.20.6 -l
Logging in to iface: default, target: iqn.2006-01.com.openfiler:orasvr01vg-vol03, portal: 200.200.20.6,3260
Login to [iface: default, target: iqn.2006-01.com.openfiler:orasvr01vg-vol03, portal: 200.200.20.6,3260] successful.

[root@orasvr01 ~]# iscsiadm -m node -T iqn.2006-01.com.openfiler:orasvr01vg-vol04 -p 200.200.20.6 -l
Logging in to iface: default, target: iqn.2006-01.com.openfiler:orasvr01vg-vol04, portal: 200.200.20.6,3260
Login to [iface: default, target: iqn.2006-01.com.openfiler:orasvr01vg-vol04, portal: 200.200.20.6,3260] successful.

[root@orasvr01 ~]# iscsiadm -m node -T iqn.2006-01.com.openfiler:orasvr01vg-vol05 -p 200.200.20.6 -l
Logging in to iface: default, target: iqn.2006-01.com.openfiler:orasvr01vg-vol05, portal: 200.200.20.6,3260
Login to [iface: default, target: iqn.2006-01.com.openfiler:orasvr01vg-vol05, portal: 200.200.20.6,3260] successful.

[root@orasvr01 ~]# iscsiadm -m node -T iqn.2006-01.com.openfiler:orasvr01vg-vol06 -p 200.200.20.6 -l
Logging in to iface: default, target: iqn.2006-01.com.openfiler:orasvr01vg-vol06, portal: 200.200.20.6,3260
Login to [iface: default, target: iqn.2006-01.com.openfiler:orasvr01vg-vol06, portal: 200.200.20.6,3260] successful.

[root@orasvr01 ~]# iscsiadm -m node -T iqn.2006-01.com.openfiler:orasvr01vg-vol13 -p 200.200.20.6 -l
Logging in to iface: default, target: iqn.2006-01.com.openfiler:orasvr01vg-vol13, portal: 200.200.20.6,3260
Login to [iface: default, target: iqn.2006-01.com.openfiler:orasvr01vg-vol13, portal: 200.200.20.6,3260] successful.

[root@orasvr01 ~]# iscsiadm -m node -T iqn.2006-01.com.openfiler:orasvr01vg-vol15 -p 200.200.20.6 -l
Logging in to iface: default, target: iqn.2006-01.com.openfiler:orasvr01vg-vol15, portal: 200.200.20.6,3260
Login to [iface: default, target: iqn.2006-01.com.openfiler:orasvr01vg-vol15, portal: 200.200.20.6,3260] successful.

[root@orasvr01 ~]# iscsiadm -m node -T iqn.2006-01.com.openfiler:orasvr01vg-vol16 -p 200.200.20.6 -l
Logging in to iface: default, target: iqn.2006-01.com.openfiler:orasvr01vg-vol16, portal: 200.200.20.6,3260
Login to [iface: default, target: iqn.2006-01.com.openfiler:orasvr01vg-vol16, portal: 200.200.20.6,3260] successful.

[root@orasvr01 ~]# iscsiadm -m node -T iqn.2006-01.com.openfiler:orasvr01vg-vol17 -p 200.200.20.6 -l
Logging in to iface: default, target: iqn.2006-01.com.openfiler:orasvr01vg-vol17, portal: 200.200.20.6,3260
Login to [iface: default, target: iqn.2006-01.com.openfiler:orasvr01vg-vol17, portal: 200.200.20.6,3260] successful.

This has the effect of Oracle Linux creating disk devices for each iSCSI target. We can see the initial iSCSI target to disk device mapping here:

[root@orasvr01 ~]# ls -l /dev/disk/by-path | grep iscsi
lrwxrwxrwx 1 root root 9 Jul  4 18:48 ip-192.168.1.6:3260-iscsi-iqn.2006-01.com.openfiler:orasvr01vg-vol01-lun-0 -> ../../sda
lrwxrwxrwx 1 root root 9 Jul  4 18:49 ip-192.168.1.6:3260-iscsi-iqn.2006-01.com.openfiler:orasvr01vg-vol02-lun-0 -> ../../sdb
lrwxrwxrwx 1 root root 9 Jul  4 18:49 ip-192.168.1.6:3260-iscsi-iqn.2006-01.com.openfiler:orasvr01vg-vol03-lun-0 -> ../../sdc
lrwxrwxrwx 1 root root 9 Jul  4 18:49 ip-192.168.1.6:3260-iscsi-iqn.2006-01.com.openfiler:orasvr01vg-vol04-lun-0 -> ../../sdd
lrwxrwxrwx 1 root root 9 Jul  4 18:58 ip-192.168.1.6:3260-iscsi-iqn.2006-01.com.openfiler:orasvr01vg-vol05-lun-0 -> ../../sde
lrwxrwxrwx 1 root root 9 Jul  4 18:58 ip-192.168.1.6:3260-iscsi-iqn.2006-01.com.openfiler:orasvr01vg-vol06-lun-0 -> ../../sdf
lrwxrwxrwx 1 root root 9 Jul  4 18:59 ip-192.168.1.6:3260-iscsi-iqn.2006-01.com.openfiler:orasvr01vg-vol13-lun-0 -> ../../sdg
lrwxrwxrwx 1 root root 9 Jul  4 18:59 ip-192.168.1.6:3260-iscsi-iqn.2006-01.com.openfiler:orasvr01vg-vol15-lun-0 -> ../../sdh
lrwxrwxrwx 1 root root 9 Jul  4 18:59 ip-192.168.1.6:3260-iscsi-iqn.2006-01.com.openfiler:orasvr01vg-vol16-lun-0 -> ../../sdi
lrwxrwxrwx 1 root root 9 Jul  4 18:59 ip-192.168.1.6:3260-iscsi-iqn.2006-01.com.openfiler:orasvr01vg-vol17-lun-0 -> ../../sdj

Next, we need to configure automatic iSCSI client login so the server will log into the iSCSI targets each time the system is started or rebooted:

[root@orasvr01 ~]# iscsiadm -m node -T iqn.2006-01.com.openfiler:orasvr01vg-vol01 -p 192.168.1.6 --op update -n node.startup -v automatic

[root@orasvr01 ~]# iscsiadm -m node -T iqn.2006-01.com.openfiler:orasvr01vg-vol02 -p 192.168.1.6 --op update -n node.startup -v automatic

[root@orasvr01 ~]# iscsiadm -m node -T iqn.2006-01.com.openfiler:orasvr01vg-vol03 -p 192.168.1.6 --op update -n node.startup -v automatic

[root@orasvr01 ~]# iscsiadm -m node -T iqn.2006-01.com.openfiler:orasvr01vg-vol04 -p 192.168.1.6 --op update -n node.startup -v automatic

[root@orasvr01 ~]# iscsiadm -m node -T iqn.2006-01.com.openfiler:orasvr01vg-vol05 -p 192.168.1.6 --op update -n node.startup -v automatic

[root@orasvr01 ~]# iscsiadm -m node -T iqn.2006-01.com.openfiler:orasvr01vg-vol06 -p 192.168.1.6 --op update -n node.startup -v automatic

[root@orasvr01 ~]# iscsiadm -m node -T iqn.2006-01.com.openfiler:orasvr01vg-vol13 -p 192.168.1.6 --op update -n node.startup -v automatic

[root@orasvr01 ~]# iscsiadm -m node -T iqn.2006-01.com.openfiler:orasvr01vg-vol15 -p 192.168.1.6 --op update -n node.startup -v automatic

[root@orasvr01 ~]# iscsiadm -m node -T iqn.2006-01.com.openfiler:orasvr01vg-vol16 -p 192.168.1.6 --op update -n node.startup -v automatic

[root@orasvr01 ~]# iscsiadm -m node -T iqn.2006-01.com.openfiler:orasvr01vg-vol17 -p 192.168.1.6 --op update -n node.startup -v automatic

Task #11: Partition Disks.

For now, I’ll only partition the six 20 GB disk devices. The basic sequence of steps would be the same for each. Here’s how to do the first one:

[root@orasvr01 ~]# fdisk /dev/sda
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Command (m for help): n
Partition type:    
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-41943039, default 2048): 
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-41943039, default 41943039): 
Using default value 41943039
Partition 1 of type Linux and of size 20 GiB is set

Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.

This is what you end up with:

[root@orasvr01 ~]# fdisk –l /dev/sda
Disk /dev/sda: 21.5 GB, 21474836480 bytes, 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0xfba01026

Device Boot      Start         End      Blocks   Id  System
/dev/sda1         2048    41943039    20970496   83  Linux

Task #12: Configure Persistent Disk Device Names for iSCSI Targets.

Each time the server is booted, the iSCSI targets could be assigned a different device name. That’s a problem because it means your database files could be seen to change file system mount points. To ensure a given iSCSI target always maps to the same disk device, we can use udev rules. This has changed a little between Oracle Linux 6 and Oracle Linux 7.

Step 1 is to obtain the unique iSCSI id of each disk device we need a udev rule for. To do that, use the scsi_id command. It’s location has changed in OL7 and its path is no longer part of the root user’s default path. Here’s a quick alias and usage of the command:

[root@orasvr01 ~]# alias scsi_id='/usr/lib/udev/scsi_id'

[root@orasvr01 ~]# scsi_id -g -u -d /dev/sda
14f504e46494c455236547044684f2d336333412d51514561

[root@orasvr01 ~]# scsi_id -g -u -d /dev/sdb
14f504e46494c45526d31426172632d647932672d3450636f

[root@orasvr01 ~]# scsi_id -g -u -d /dev/sdc
14f504e46494c45523647754338332d563959522d4343746c

[root@orasvr01 ~]# scsi_id -g -u -d /dev/sdd
14f504e46494c4552716c344377502d474541532d4b56704d

[root@orasvr01 ~]# scsi_id -g -u -d /dev/sde
14f504e46494c4552736b3773624e2d4932594b2d76426b65

[root@orasvr01 ~]# scsi_id -g -u -d /dev/sdf
14f504e46494c45523471744464762d3532784b2d37314169

Step 2 is to create a udev rules script in /etc/udev/rules.d directory. The script can be called anything you like so long as it starts with a number and ends with “.rules”. It’s always important to name your script something meaningful. Here’s my file with the relevant syntax, one line per iSCSI target:

[root@orasvr01 rules.d]# ls -l
-rw-r--r-- 1 root root 1332 Jul  5 11:16 99-openfilerdevices.rules

[root@orasvr01 rules.d]# cat 99-openfilerdevices.rules
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="14f504e46494c455236547044684f2d336333412d51514561", SYMLINK+="orasvr01vg-vol01", OWNER="root", GROUP="disk", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="14f504e46494c45526d31426172632d647932672d3450636f", SYMLINK+="orasvr01vg-vol02", OWNER="root", GROUP="disk", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="14f504e46494c45523647754338332d563959522d4343746c", SYMLINK+="orasvr01vg-vol03", OWNER="root", GROUP="disk", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="14f504e46494c4552716c344377502d474541532d4b56704d", SYMLINK+="orasvr01vg-vol04", OWNER="root", GROUP="disk", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="14f504e46494c4552736b3773624e2d4932594b2d76426b65", SYMLINK+="orasvr01vg-vol05", OWNER="root", GROUP="disk", MODE="0660"
KERNEL=="sd?1", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent", RESULT=="14f504e46494c45523471744464762d3532784b2d37314169", SYMLINK+="orasvr01vg-vol06", OWNER="root", GROUP="disk", MODE="0660"

A little explanation:

Parameter Value Comment
KERNEL sd?1 Defaults to a wildcard pattern match for the disk device
SUBSYSTEM block /dev/sd?1 are block devices
PROGRAM /usr/lib/udev/scsi_id … Path to the scsi_id executable
RESULT (iSCSI ID) Unique iSCSI identifier returned by scsi_id
SYMLINK+ (Name) The symbolic link name which points to the disk device
OWNER root By default the root user owns disk devices
GROUP disk By default the OS group is disk
MODE 0660 Default permissions mask for the disk device

Step 3 is to test the resolution of each line in the rules file. This is important because running a test actually creates the symbolic link. We already know what the current iSCSI target to disk device mappings are:

iSCSI Target Disk Device
iqn.2006-01.com.openfiler:orasvr01vg-vol01 /dev/sda
iqn.2006-01.com.openfiler:orasvr01vg-vol02 /dev/sdb
iqn.2006-01.com.openfiler:orasvr01vg-vol03 /dev/sdc
iqn.2006-01.com.openfiler:orasvr01vg-vol04 /dev/sdd
iqn.2006-01.com.openfiler:orasvr01vg-vol05 /dev/sde
iqn.2006-01.com.openfiler:orasvr01vg-vol06 /dev/sdf

Taking /dev/sda as an example:

[root@orasvr01 ~]# ls -l /dev | grep sda
brw-rw---- 1 root disk      8,   0 Jul  4 19:35 sda
brw-rw---- 1 root disk      8,   1 Jul  4 19:35 sda1

We see they are block devices (b), their permissions mask is 0660 (rw-rw—-), they’re owned by root and belong to the group, disk.

Let’s run the test for /dev/sda1:

[root@orasvr01 ~]# ls -l /dev | grep orasvr01
(no output)

[root@orasvr01 ~]# udevadm test /block/sda/sda1

The output is quite verbose, but can be seen in its entirety here. Once the test completes, check to see if a symbolic link has shown up:

[root@orasvr01 ~]# ls -l /dev | grep orasvr01
lrwxrwxrwx 1 root root           4 Jul  5 12:08 orasvr01vg-vol01 -> sda1

Hurrah! By referencing the symbolic link and trusting that it always points to the correct disk device we’re all set to either build file systems on orasvr01 or create ASM Disks on orasvr02. Don’t forget to run the test for all the other disk devices to ensure their symbolic links get created.

Task #13a: Create File Systems (orasvr01).

First we need to build the /u01 file system whose storage is coming from the VM_Filesystems_Repo storage repository. Linux disk devices which come from OVM follow the naming convention /dev/xvd<letter>, where letter starts with a, then b and so on. The /dev/xvda disk has already been used for the Linux OS, so we should have /dev/xvdb waiting for us. Let’s check:

[root@orasvr01 ~]# fdisk -l /dev/xvdb
Disk /dev/xvdb: 64.4 GB, 64424509440 bytes, 125829120 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Let’s create a primary partition, then build the file system:

[root@orasvr01 ~]# fdisk /dev/xvdb
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x49a8eb2a.

Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-125829119, default 2048): 
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-125829119, default 125829119): 
Using default value 125829119
Partition 1 of type Linux and of size 60 GiB is set

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

[root@orasvr01 ~]# mkfs -t ext4 -m 0 /dev/xvdb1
mke2fs 1.42.9 (28-Dec-2013)
Discarding device blocks: done                            
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
3932160 inodes, 15728384 blocks
0 blocks (0.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=2164260864
480 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks: 
         32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
         4096000, 7962624, 11239424

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done 

Let’s go ahead and build file systems using the symbolic links which point to the 6 disk devices whose storage is coming from Openfiler:

[root@orasvr01 ~]# mkfs -t ext4 -m 0 /dev/orasvr01vg-vol01
[root@orasvr01 ~]# mkfs -t ext4 -m 0 /dev/orasvr01vg-vol02
[root@orasvr01 ~]# mkfs -t ext4 -m 0 /dev/orasvr01vg-vol03
[root@orasvr01 ~]# mkfs -t ext4 -m 0 /dev/orasvr01vg-vol04
[root@orasvr01 ~]# mkfs -t ext4 -m 0 /dev/orasvr01vg-vol05
[root@orasvr01 ~]# mkfs -t ext4 -m 0 /dev/orasvr01vg-vol06

Next, create the mount point directories for these file systems:

[root@orasvr01 ~]# cd /
[root@orasvr01 /]# mkdir /u01 /u02 /u03 /u04 /u05 /u06 /u07

[root@orasvr01 /]# ls -l
lrwxrwxrwx.   1 root root    7 Jun 29 11:35 bin -> usr/bin
dr-xr-xr-x.   4 root root 4096 Jun 29 12:40 boot
drwxr-xr-x   20 root root 3780 Jul  5 15:05 dev
drwxr-xr-x.  88 root root 8192 Jul  4 17:44 etc
drwxr-xr-x.   5 root root   44 Jul  4 13:55 home
lrwxrwxrwx.   1 root root    7 Jun 29 11:35 lib -> usr/lib
lrwxrwxrwx.   1 root root    9 Jun 29 11:35 lib64 -> usr/lib64
drwxr-xr-x.   2 root root    6 Apr 10  2018 media
drwxr-xr-x.   2 root root    6 Apr 10  2018 mnt
drwxr-xr-x.   3 root root   16 Jun 29 11:37 opt
dr-xr-xr-x  164 root root    0 Jul  1 12:05 proc
dr-xr-x---.   5 root root 4096 Jul  5 10:49 root
drwxr-xr-x   28 root root  820 Jul  4 17:44 run
lrwxrwxrwx.   1 root root    8 Jun 29 11:35 sbin -> usr/sbin
drwxr-xr-x.   2 root root    6 Apr 10  2018 srv
dr-xr-xr-x   13 root root    0 Jul  4 15:35 sys
drwxrwxrwt.   7 root root 4096 Jul  5 03:10 tmp
drwxr-xr-x    2 root root    6 Jul  5 15:07 u01
drwxr-xr-x    2 root root    6 Jul  5 15:07 u02
drwxr-xr-x    2 root root    6 Jul  5 15:07 u03
drwxr-xr-x    2 root root    6 Jul  5 15:07 u04
drwxr-xr-x    2 root root    6 Jul  5 15:07 u05
drwxr-xr-x    2 root root    6 Jul  5 15:07 u06
drwxr-xr-x    2 root root    6 Jul  5 15:07 u07
drwxr-xr-x.  13 root root 4096 Jun 29 11:35 usr
drwxr-xr-x.  20 root root 4096 Jun 29 12:30 var

Next, add the relevant entries to the /etc/fstab file:

/dev/xvdb1              /u01                    ext4    defaults        0 0
/dev/orasvr01vg-vol01   /u02                    ext4    defaults        0 0
/dev/orasvr01vg-vol02   /u03                    ext4    defaults        0 0
/dev/orasvr01vg-vol03   /u04                    ext4    defaults        0 0
/dev/orasvr01vg-vol04   /u05                    ext4    defaults        0 0
/dev/orasvr01vg-vol05   /u06                    ext4    defaults        0 0
/dev/orasvr01vg-vol06   /u07                    ext4    defaults        0 0

Finally, mount all the file systems and check they’re available:

[root@orasvr01 /]# mount –a
[root@orasvr01 /]# df -h
Filesystem           Size  Used Avail Use% Mounted on
devtmpfs             3.9G     0  3.9G   0% /dev
tmpfs                3.9G     0  3.9G   0% /dev/shm
tmpfs                3.9G  8.6M  3.9G   1% /run
tmpfs                3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/mapper/ol-root   35G  1.8G   34G   5% /
/dev/xvda1          1014M  235M  780M  24% /boot
tmpfs                797M     0  797M   0% /run/user/0
/dev/xvdb1            59G   53M   59G   1% /u01
/dev/sda1             20G   45M   20G   1% /u02
/dev/sdb1             20G   45M   20G   1% /u03
/dev/sdc1             20G   45M   20G   1% /u04
/dev/sdd1             20G   45M   20G   1% /u05
/dev/sde1             20G   45M   20G   1% /u06
/dev/sdf1             20G   45M   20G   1% /u07

That’s it! We’re now ready to copy the Oracle Database code sets to /u01, install them and build databases using the storage in /u02 through /u07 on orasvr01.

Note, if you install Oracle Database 11g Release 11.2.0.4 on Oracle Linux 7, check this first!

Task #13b: Create ASM Disks (orasvr02).

As we know, each time the server is booted the iSCSI targets could be assigned a different device name. So the trick is to ensure the device name is persistent. That way Oracle will know which files are where. This can be done using udev rules and/or using Oracle’s ASMLib. Been there, done that, works great. There is a third way, however. Beginning with Oracle Database 12c Release 1 (12.1.0.2), Oracle provided the Oracle ASM Filter Driver (ASMFD). This is now Oracle’s recommended way to configure storage for ASM.

According to the Oracle ASMFD documentation, ASMFD “simplifies the configuration and management of disk devices by eliminating the need to rebind disk devices used with Oracle ASM each time the system is restarted”. So device name peristence then. It is further claimined that ASMFD is a kernel module which resides in the I/O path of the ASM Disks. Hence, its function is also to validate I/O requests to ASM Disks, filtering out invalid I/O. Presumably that means if you attempted to interfere with a system’s ASM Disks directly, the ASM Filter Driver would prevent you from doing so. Could be an interesting experiment when you have a spare moment. Don’t try it on a production system though, right?

Anyway, let’s try out the ASM Filter Driver. The driver can be installed and configured as part of an Oracle Grid Infrastructure installation. Pretty handy. Since the rules of the game are you have to use an ASM instance whose version is equal to or exceeds that of your database, let’s go with Oracle Grid Infrastructure 19c Release 3. Before we run the installer, let’s review the iSCSI targets and disk devices the operating system can currently see:

iSCSI Target OS Device Name (GB) Candidate ASM Disk
openfiler:orasvr02vg-vol07-lun-0 /dev/sdj (20) DATA_000
openfiler:orasvr02vg-vol08-lun-0 /dev/sdi (20) DATA_001
openfiler:orasvr02vg-vol09-lun-0 /dev/sdh (20) DATA_002
openfiler:orasvr02vg-vol10-lun-0 /dev/sdg (20) RECO_000
openfiler:orasvr02vg-vol11-lun-0 /dev/sdf (20) RECO_001
openfiler:orasvr02vg-vol12-lun-0 /dev/sde (20) RECO_002
openfiler:orasvr02vg-vol14-lun-0 /dev/sdd (10) REDO_000

I’m not going to define any udev rules to ensure ASMFD takes charge of device persistence. Previously we’ve used ASMLib’s oracleasm command to create candidate ASM Disks. When using ASMFD there’s a similar procedure to follow. We’ll get to that in a moment since the utility we need to use is contained in the Grid Infrastructure code set which we haven’t unzipped yet.

As is usual, there are a bunch of prerequisites to check before we get to install the Grid Infrastructure software. Since we’ll be performing a stand alone installation, the RAC and Clusterware prerequisites will not apply. Despite that, there are still a number of things to check. The full Oracle Grid Infrastructure Installation Checklist is here.

Just for fun, let’s check a few of the pre-installation items.

Pre-install Check #1: At least 8 GB of RAM.

[root@orasvr02 ~]# grep MemTotal /proc/meminfo
MemTotal:       16412260 kB

Pre-install Check #2: Oracle Linux 7.4 with UEK 4:4.1.12-124.19.2.el7uek.x86_64 or later.

[root@orasvr02 ~]# cat /etc/oracle-release
Oracle Linux Server release 7.6

[root@orasvr02 ~]# uname -a
Linux orasvr02.mynet.com 4.14.35-1902.2.0.el7uek.x86_64 #2 SMP Fri Jun 14 21:15:44 PDT 2019 x86_64 x86_64 x86_64 GNU/Linux

There is a major gotcha with this one which we’ll get to later. Stay tuned.

Pre-install Check #3: Oracle Preinstallation RPM for Oracle Linux.

[root@orasvr02 ~]# rpm -qa | grep pre
oracle-database-preinstall-19c-1.0-1.el7.x86_64

Note, when the preinstallation package is installed, a file called /etc/security/limits.d/oracle-database-preinstall-19c.conf gets created. The values in this file override settings in the standard /etc/security/limits.conf file. Also note, the package only defines values for the oracle user. Since we’ll be using the grid user to install Grid Infrastructure, you’ll need to duplicate all the oracle user entries for the grid user.

Pre-install Check #4: Disable transparent HugePages.

[root@orasvr02 ~]# cat /sys/kernel/mm/transparent_hugepage/enabled
always madvise [never]

Oracle recommends the use of HugePages on Linux. For additional information on those, go here. (coming soon)

Pre-install Check #5: Swap Space allocation relative to RAM

For 16 GB of RAM, we should have 16 GB of swap space:

[root@orasvr02 ~]# swapon --show
NAME      TYPE      SIZE USED PRIO
/dev/dm-1 partition   4G   0B   -2

Since we only have 4 GB of swap, we need another 12 GB. This can be done by adding a swap file.

Pre-install Check #6: Disk I/O Scheduler.

Oracle recommends the Deadline I/O scheduler for the best performance with ASM Disks. However, the Oracle documentation also says this:

“On some virtual environments (VM) and special devices such as fast storage devices, the output of the above command may be none. The operating system or VM bypasses the kernel I/O scheduling and submits all I/O requests directly to the device. Do not change the I/O Scheduler settings on such environments.”

In our configuration we are using VMs, but the storage is actually coming from a NAS (Openfiler). So let’s check which I/O scheduler Oracle Linux is using for the storage coming from the NAS:

[root@orasvr02 ~]# for disk in d e f g h i j
> do
> echo "Checking /dev/sd$disk: `cat /sys/block/sd$disk/queue/scheduler`"
> done

Checking /dev/sdd: noop [deadline] cfq 
Checking /dev/sde: noop [deadline] cfq 
Checking /dev/sdf: noop [deadline] cfq 
Checking /dev/sdg: noop [deadline] cfq 
Checking /dev/sdh: noop [deadline] cfq 
Checking /dev/sdi: noop [deadline] cfq 
Checking /dev/sdj: noop [deadline] cfq 

Pre-install Check #7: Creating Users, Groups and Paths.

[root@orasvr02 ~]# fgrep grid /etc/passwd
grid:x:1001:1000::/home/grid:/bin/bash

[grid@orasvr02 ~]$ id
uid=1001(grid) gid=1000(oinstall) groups=1000(oinstall),1001(dba),1006(racdba),1007(asmdba),1008(asmadmin),1009(asmoper)

We need 3 paths for the Grid Infrastructre installation. A grid user ORACLE_BASE, an Oracle Inventory and a grid user ORACLE_HOME. Note, if this is the first Oracle software installtion on the server, then the Oracle Inventory directory must be owned by the software owner, which in our case is the grid user:

[root@orasvr02 ~]# ls -l /u01/app
drwxr-xr-x 3 grid   oinstall 4096 Nov  6 15:54 grid
drwxrwxr-x 2 grid   oinstall 4096 Nov  5 12:18 oraInventory

[root@orasvr02 ~]# ls -l /u01/app/oracle/product/19.3.0/
drwxr-xr-x 67 grid   oinstall 4096 Nov  6 15:54 grid

Pre-install Check #8: Configuring Candidate ASM Disks.

The Grid Infrastructure installation will attempt to create an initial ASM Diskgroup for which it will need a candidate ASM Disk. The utility used to create candidate ASM Disks is contained within the Grid Infrastructure code set, so we need to copy the downloaded Grid Infrastructure zip file to the grid user’s ORACLE_HOME directory and unzip it:

[grid@orasvr02 ~]$ cp /u01/MEDIA/GI_19.3.0/LINUX.X64_193000_grid_home.zip /u01/app/oracle/product/19.3.0/grid

[grid@orasvr02 ~]$ cd /u01/app/oracle/product/19.3.0/grid
[grid@orasvr02 grid]$ unzip LINUX.X64_193000_grid_home.zip
...

Next, login as root, set your environment then use the asmcmd command to label the disks, thus turning them into candidate ASM Disks:

[grid@orasvr02 grid]$ su - 
Password: 
Last login: Thu Nov  7 13:57:02 CST 2019 on pts/1

[root@orasvr02 ~]# cd /u01/app/grid
[root@orasvr02 grid]# export ORACLE_BASE=`pwd`

[root@orasvr02 grid]# cd /u01/app/oracle/product/19.3.0/grid
[root@orasvr02 grid]# export ORACLE_HOME=`pwd`

[root@orasvr02 grid]# env | grep ORA
ORACLE_BASE=/u01/app/grid
ORACLE_HOME=/u01/app/oracle/product/19.3.0/grid

[root@orasvr02 grid]# cd bin

The disk devices we’ll be using for ASM on this server are /dev/sdd through /dev/sdj (see above). I have created a single primary partition for each disk device. Let’s quickly check what the output looks like when you check the label of a disk which has not yet been labeled:

[root@orasvr02 bin]# for disk in d e f g h i j
> do
> echo "Checking ASM Label of /dev/sd$disk: `./asmcmd afd_lslbl /dev/sd$disk`"
> done 

Checking ASM Label of /dev/sdd: No devices to be scanned.
Checking ASM Label of /dev/sde: No devices to be scanned.
Checking ASM Label of /dev/sdf: No devices to be scanned.
Checking ASM Label of /dev/sdg: No devices to be scanned.
Checking ASM Label of /dev/sdh: No devices to be scanned.
Checking ASM Label of /dev/sdi: No devices to be scanned.
Checking ASM Label of /dev/sdj: No devices to be scanned.

So let’s crack on and label those disks (these commands produce no output):

[root@orasvr02 bin]# ./asmcmd afd_label DATA_0000 /dev/sdj --init
[root@orasvr02 bin]# ./asmcmd afd_label DATA_0001 /dev/sdi --init
[root@orasvr02 bin]# ./asmcmd afd_label DATA_0002 /dev/sdh --init
[root@orasvr02 bin]# ./asmcmd afd_label RECO_0000 /dev/sdg --init
[root@orasvr02 bin]# ./asmcmd afd_label RECO_0001 /dev/sdf --init
[root@orasvr02 bin]# ./asmcmd afd_label RECO_0002 /dev/sde --init
[root@orasvr02 bin]# ./asmcmd afd_label REDO_0000 /dev/sdd --init

Now let’s check the labels are correct:

[root@orasvr02 bin]# for disk in d e f g h i j
> do
> echo "Checking ASM Label of /dev/sd$disk: `./asmcmd afd_lslbl /dev/sd$disk`"
> done 

Checking ASM Label of /dev/sdd: ------------------------------------------------
Label                     Duplicate  Path
================================================================================
REDO_0000                             /dev/sdd
Checking ASM Label of /dev/sde: ------------------------------------------------
Label                     Duplicate  Path
================================================================================ 
RECO_0002                             /dev/sde
Checking ASM Label of /dev/sdf: ------------------------------------------------
Label                     Duplicate  Path
================================================================================ 
RECO_0001                             /dev/sdf
Checking ASM Label of /dev/sdg: ------------------------------------------------
Label                     Duplicate  Path
================================================================================ 
RECO_0000                             /dev/sdg
Checking ASM Label of /dev/sdh: ------------------------------------------------
Label                     Duplicate  Path
================================================================================ 
DATA_0002                             /dev/sdh
Checking ASM Label of /dev/sdi: ------------------------------------------------
Label                     Duplicate  Path
================================================================================ 
DATA_0001                             /dev/sdi
Checking ASM Label of /dev/sdj: ------------------------------------------------
Label                     Duplicate  Path
================================================================================ 
DATA_0000                             /dev/sdj

In addition, labeling the disks creates files in /dev/oracleafd/disks:

[root@orasvr02 ~]# ls -l /dev/oracleafd/disks
-rw-rw-r-- 1 grid oinstall 32 Nov  8 11:43 DATA_0000
-rw-rw-r-- 1 grid oinstall 32 Nov  8 11:43 DATA_0001
-rw-rw-r-- 1 grid oinstall 32 Nov  8 11:44 DATA_0002
-rw-rw-r-- 1 grid oinstall 32 Nov  8 11:44 RECO_0000
-rw-rw-r-- 1 grid oinstall 32 Nov  8 11:44 RECO_0001
-rw-rw-r-- 1 grid oinstall 32 Nov  8 11:44 RECO_0002
-rw-rw-r-- 1 grid oinstall 32 Nov  8 11:44 REDO_0000 

Install Grid Infrastructure 19c Release 3.

Now we’re ready to run the installation, but before we do be aware of a significant limitation of Oracle’s UEK. I believe that Oracle Linux 7.6 UEK and above does not (yet) support ASMFD. I ran into this issue and kept getting these errors when attempting to navigate past Step #2 of the installation (i.e. the ASM Diskgroup creation and ASMFD configuration):

INS-41223: ASM Filter Driver is not supported on this platform.

AFD-620: AFD is not supported on this operating system version: '4.14.35-1902.2.0.el7uek.x86_64' 

So Oracle’s stuff doesn’t work on Oracle’s stuff. Lovely. To get around this, I booted to a different kernel. To change the default boot kernel, go here. Once you’re running a supported kernel, check to see Oracle agrees using a couple of utilities:

[root@orasvr02 ~]# cd /u01/app/grid
[root@orasvr02 grid]# export ORACLE_BASE=pwd

[root@orasvr02 grid]# cd ../oracle/product/19.3.0/grid
[root@orasvr02 grid]# export ORACLE_HOME=pwd

[root@orasvr02 grid]# cd bin

[root@orasvr02 bin]# ./afdroot version_check
AFD-616: Valid AFD distribution media detected at: '/u01/app/oracle/product/19.3.0/grid/usm/install/Oracle/EL7/x86_64/3.10.0-862/3.10.0-862-x86_64/bin'

[root@orasvr02 bin]# ./afddriverstate -orahome /u01/app/oracle/product/19.3.0/grid version
AFD-9325:     Driver OS kernel version = 3.10.0-862.el7.x86_64.
AFD-9326:     Driver build number = 190222.
AFD-9212:     Driver build version = 19.0.0.0.0.
AFD-9547:     Driver available build number = 190222.
AFD-9548:     Driver available build version = 19.0.0.0.0.

[root@orasvr02 bin]# ./afddriverstate -orahome /u01/app/oracle/product/19.3.0/grid supported
AFD-9200: Supported

Assuming you’re running a kernel which plays nicely with ASMFD, let’s start the installtion.Login as the grid user, set your environment then run the installer script, gridSetup.sh:

[grid@orasvr02 ~]$ export ORACLE_BASE=/u01/app/grid
[grid@orasvr02 ~]$ export ORACLE_HOME=/u01/app/oracle/product/19.3.0/grid
[grid@orasvr02 ~]$ export DISPLAY=<your-workstation-or-IP-address>:0.0
[grid@orasvr02 ~]$ cd $ORACLE_HOME
[grid@orasvr02 grid]$ ls -l gridSetup.sh
-rwxr-x--- 1 grid oinstall 3294 Mar  8  2017 gridSetup.sh

[grid@orasvr02 grid]$ ./gridSetup.sh
Select the Standalone Server (Oracle Restart) option then click Next
Click External Redundancy, 3 disks for the DATA ASM Diskgroup & checkbox to configure ASM Filter Driver , then Next
You might be looking at this screen for a while – if your kernel is incompatible with ASMFD this is where you find out!
Enter a password for the ASM SYS and ASMSNMP accounts then click Next
Complete this screen if you already have Cloud Control setup otherwise just click Next
If you’ve setup your OS groups correctly then click Next
This screen picks up the values for ORACLE_BASE & ORACLE_HOME from your OS environment – Click Next
Click Next
We will run the root scripts in a separate session – Click Next
The installer runs its pre-req checks and if all is well takes you to the next screen
Check the options and if all looks good click Install
Sit back and wait for the root scripts
Run the root scripts in a separate terminal session
[root@orasvr02 ~]# /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.

Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.

[root@orasvr02 ~]# /u01/app/oracle/product/19.3.0/grid/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /u01/app/oracle/product/19.3.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]: 
    Copying dbhome to /usr/local/bin …
    Copying oraenv to /usr/local/bin …
    Copying coraenv to /usr/local/bin …

Creating /etc/oratab file…
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/oracle/product/19.3.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
   /u01/app/grid/crsdata/orasvr02/crsconfig/roothas_2019-11-08_12-05-49AM.log
LOCAL ADD MODE 
Creating OCR keys for user 'grid', privgrp 'oinstall'..
Operation successful.
LOCAL ONLY MODE 
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4664: Node orasvr02 successfully pinned.
2019/11/08 12:11:54 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'

orasvr02 2019/11/08 12:20:56  /u01/app/grid/crsdata/orasvr02/olr/backup_20191108_122056.olr 724960844     
2019/11/08 12:22:55 CLSRSC-327: Successfully configured Oracle Restart for a standalone server
Clicking OK in the run root scripts window continues the installation and configuration steps
If all went well you see this screen – click Close

There are some post installation tasks that you might want to pay attention to. Those are documented here. Before finishing up with the ASM Diskgroup configuration, let’s check the status of the ASM Filter Driver now that the installation has completed (this command took ages to return output on my system – YMMV):

[root@orasvr02 ~]# cd /u01/app/oracle/product/19.3.0/grid/bin
[root@orasvr02 bin]# ./asmcmd afd_state
ASMCMD-9526: The AFD state is 'LOADED' and filtering is 'ENABLED' on host 'orasvr02.mynet.com'

Finally, create the remaining ASM Diskgroups in preparation for creating databases. First, check what the ASM Diskgroup configuration looks like so far:

[grid@orasvr02 ~]$ sqlplus / as sysasm
 
SQL> select dg.name "Diskgroup Name", d.name "Disk Name", d.label "Disk Label", d.path "Disk Path"
     from v$asm_diskgroup dg, v$asm_disk d
     where dg.group_number(+) = d.group_number
     order by dg.name, d.name
     /

Diskgroup Name  Disk Name       Disk Label      Disk Path
--------------- --------------- --------------- --------------- 
DATA            DATA_0000       DATA_0000       AFD:DATA_0000
DATA            DATA_0001       DATA_0001       AFD:DATA_0001
DATA            DATA_0002       DATA_0002       AFD:DATA_0002
                                REDO_0000       AFD:REDO_0000
                                RECO_0001       AFD:RECO_0001
                                RECO_0000       AFD:RECO_0000
                                RECO_0002       AFD:RECO_0002
 7 rows selected.

SQL> save dg_query
Created file dg_query.sql

Create the RECO and REDO ASM Diskgroups, then re-check the configuration:

SQL> create diskgroup reco 
     external redundancy
     disk 'AFD:RECO_0000', 'AFD:RECO_0001', 'AFD:RECO_0002';

Diskgroup created.

SQL> create diskgroup redo
     external redundancy
     disk 'AFD:REDO_0000';

Diskgroup created.

SQL> @dg_query

Diskgroup Name  Disk Name       Disk Label      Disk Path
--------------- --------------- --------------- ---------------  
DATA            DATA_0000       DATA_0000       AFD:DATA_0000
DATA            DATA_0001       DATA_0001       AFD:DATA_0001
DATA            DATA_0002       DATA_0002       AFD:DATA_0002
RECO            RECO_0000       RECO_0000       AFD:RECO_0000
RECO            RECO_0001       RECO_0001       AFD:RECO_0001
RECO            RECO_0002       RECO_0002       AFD:RECO_0002
REDO            REDO_0000       REDO_0000       AFD:REDO_0000

7 rows selected.

That’s it! We’re now ready to copy the Oracle Database code sets to /u01, install them and build databases using the ASM Diskgroups DATA, REDO and RECO on orasvr02.

Note, if you install Oracle Database 11g Release 11.2.0.4 on Oracle Linux 7, check this first!

Build Your Own Oracle Infrastructure: Part 12 – Use Oracle Recovery Manager

Recovery Manager or RMAN as it is more commonly known, is Oracle’s proprietary backup, restore and recovery solution for Oracle databases.

In Part 12, we’ll setup RMAN, run through some common backup/restore/recovery scenarios and have a little fun with some of the more interesting things RMAN can do.

Quick links to all the tasks:

  • Task #1: Create an RMAN Catalog.
  • Task #2: Execute RMAN Backups. (coming soon)
  • Task #3: Execute RMAN Restores & Recoveries. (coming soon)
  • Task #4: More Advanced Uses of RMAN. (coming soon)
  • Task #5: RMAN Reporting. (coming soon)

Task #1: Create an RMAN Catalog.

In most cases, using an RMAN catalog is useful. It’s not strictly necessary since database backups will be recorded in the database control files anyway. Like with most things there are pros and cons. For example, if you have more than a few databases, recording backup activity in one place is more efficient and convenient. However, if your RMAN catalog database is down then your backups will fail. So you have the added overhead of protecting your RMAN catalog database. In most cases you should treat it like any other production database.

To create an RMAN catalog, just follow these 3 simple steps:

Task #1a: Create a Database for the RMAN Catalog.

Since our infrastructure is small there’s no real need to fire up a separate database to host an RMAN catalog. Instead, we’ll create another PDB within the CDB (PADMIN) which is already running the OEM Management Repository PDB (EMPDBREPOS).

Note, running more than one PDB within a CDB requires the multitenant license. Plus in a production environment, you would more likely choose to have completely separate databases for the OEM Management Repository and the RMAN catalog anyway. What we’re doing here is for demonstration/educational purposes only. 

So let’s crack on and create another PDB:

[oracle@oraemcc ~]$ . oraenv
ORACLE_SID = [oracle] ? PADMIN
The Oracle base has been set to /u01/app/oracle

[oracle@oraemcc ~]$ sqlplus / as sysdba
SQL> show con_name
CON_NAME
--------
CDB$ROOT

SQL> create pluggable database rmancat admin user rmancat_admin identified by rmancat_admin;
Pluggable database created.
 
SQL> select name,open_mode from v$pdbs;
NAME                           OPEN_MODE
------------------------------ ----------
PDB$SEED                       READ ONLY
EMPDBREPOS                     READ WRITE
RMANCAT                        MOUNTED

SQL> alter pluggable database rmancat open;
Pluggable database altered.

SQL> select name,open_mode from v$pdbs;
NAME                           OPEN_MODE
------------------------------ -----------
PDB$SEED                       READ ONLY
EMPDBREPOS                     READ WRITE
RMANCAT                        READ WRITE

Next, we’ll quickly add the relevant TNS connect string to the tnsnames.ora file:

[oracle@oraemcc ~]$ vi $ORACLE_HOME/network/admin/tnsnames.ora

RMANCAT =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = oraemcc.mynet.com)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = RMANCAT.mynet.com)
    )
  )

Task #1b: Create the Recovery Catalog Owner (RCO).

[oracle@oraemcc ~]$ sqlplus system@rmancat

SQL> create tablespace rmancat_data;
Tablespace created.

SQL> create user rco identified by rco default tablespace rmancat_data
     quota unlimited on rmancat_data;
User created.

SQL> grant create session, resource, recovery_catalog_owner to rco;
Grant succeeded.

Task #1c: Create the Recovery Catalog.

For this step, we actually log into RMAN:

[oracle@oraemcc ~]$ rman

RMAN> connect catalog rco/rco@rmancat
connected to recovery catalog database

RMAN> create catalog tablespace rmancat_data;
recovery catalog created

That’s it! The RMAN recovery catalog is now ready to use.

Install Oracle Enterprise Manager 13c Release 2

With the recent release of Oracle Enterprise Manager Cloud Control 13c Release 2 Plug-in Update 1, I felt it was time to finally say goodbye to 12c Release 5.

 

The Oracle documentation does a pretty decent job of explaining how to uninstall Oracle Enterprise Manager (OEM) Cloud Control (CC) 12c Release 5 which can be accessed here.

The only weirdness I encountered was while uninstalling the Oracle Management Agents. Everything worked successfully, but the final output of the AgentDeinstall.pl script was this:

Can't locate Carp.pm in @INC (@INC contains: 
/u01/oracle/agent/core/12.1.0.5.0/perl/lib/5.10.0/x86_64-linux-thread-multi 
/u01/oracle/agent/core/12.1.0.5.0/perl/lib/5.10.0 
/u01/oracle/agent/core/12.1.0.5.0/perl/lib/site_perl/5.10.0/x86_64-linux-thread-multi 
/u01/oracle/agent/core/12.1.0.5.0/perl/lib/site_perl/5.10.0 
/u01/oracle/agent/core/12.1.0.5.0/perl/lib/5.10.0/x86_64-linux-thread-multi 
/u01/oracle/agent/core/12.1.0.5.0/perl/lib/5.10.0/x86_64-linux-thread-multi 
/u01/oracle/agent/core/12.1.0.5.0/perl/lib/5.10.0 
/u01/oracle/agent/core/12.1.0.5.0/perl/lib/site_perl/5.10.0/x86_64-linux-thread-multi 
/u01/oracle/agent/core/12.1.0.5.0/perl/lib/site_perl/5.10.0 
/u01/oracle/agent/core/12.1.0.5.0/perl/lib/site_perl .) at 
/u01/oracle/agent/core/12.1.0.5.0/perl/lib/5.10.0/File/Path.pm line 32.

The uninstall worked OK, so I’m guessing this message is just a ‘special feature’ which we can ignore. 😏

To install OEM CC 13c R2, we will need a server. I’m going to use the same server (oraemcc) to run the Oracle Management Service (OMS) and the Management Repository Database (MRDB). Since we’ll need a little more space than we did for the OEM CC 12c R5 installation, I increased the size of the /u01 and /u02 file systems on oraemcc accordingly:

File System Old Size New Size Usage
/u01 30 GB 50 GB Oracle Software Installations
/u02 30 GB 80 GB Oracle Database Files

 

There are 4 main tasks we need to complete to get OEM CC 13c R2 up and running. Click the link you need:

Task #1: Download the Software.

Task #2: Install the Oracle Database Software (12.1.0.2).

Task #3: Create the Respository Database.

Task #4: Install OEM CC 13c R2.

 

Task #1: Download the Software.

You will need to download 3 different pieces of software. The database software, the database template and the OEM software:

Software Location
Oracle Database 12c Release 1 (12.1.0.2) Enterprise Edition for Linux x86-64 Download Page
12.1.0.2 DB Template with CDB PDB for EM 13.2.0.0 on Linux x86-64 Download Page
OEM CC 13c R2 Plug-in Update 1 for Linux x86-64 (64 bit) Download Page

We will be using a CDB with a PDB containing the Management Repository. You don’t have to set it up this way, but I’m going to use another PDB within the same CDB to host an RMAN Catalog. See Part 12 of my Build Your Own Oracle Infrastructure series for more details.

 

Task #2: Install the Oracle Database Software (12.1.0.2).

This particular task is covered here so I won’t dwell on it again.  Once the database software is installed, copy the template zip file to $ORACLE_HOME/assistants/dbca/templates and unzip it:

[oracle@oraemcc]$ pwd
/u01/app/oracle/product/12.1.0/dbhome_1/assistants/dbca/templates

[oracle@oraemcc]$ unzip 12.1.0.2.0_Database_Template_with_cdbpdb_for_EM13_2_0_0_0_Linux_x64.zip
Archive:  12.1.0.2.0_Database_Template_with_cdbpdb_for_EM13_2_0_0_0_Linux_x64.zip
  inflating: set_repo_param_12.1.0.2.0_Database_SQL_for_EM13_2_0_0_0_Large_deployment.sql 
  inflating: set_repo_param_12.1.0.2.0_Database_SQL_for_EM13_2_0_0_0_Medium_deployment.sql 
  inflating: set_repo_param_12.1.0.2.0_Database_SQL_for_EM13_2_0_0_0_Small_deployment.sql 
  inflating: shpool_12.1.0.2.0_Database_SQL_for_EM13_2_0_0_0.sql 
  inflating: 12.1.0.2.0_Database_Template_with_cdbpdb_for_EM13_2_0_0_0_Large_deployment.dbc 
  inflating: 12.1.0.2.0_Database_Template_with_cdbpdb_for_EM13_2_0_0_0_Medium_deployment.dbc 
  inflating: 12.1.0.2.0_Database_Template_with_cdbpdb_for_EM13_2_0_0_0_Small_deployment.dbc 
  inflating: 12.1.0.2.0_Database_Template_with_cdbpdb_for_EM13_2_0_0_0.dfb1 
  inflating: 12.1.0.2.0_Database_Template_with_cdbpdb_for_EM13_2_0_0_0.dfb2 
  inflating: 12.1.0.2.0_Database_Template_with_cdbpdb_for_EM13_2_0_0_0.dfb3 
  inflating: 12.1.0.2.0_Database_Template_with_cdbpdb_for_EM13_2_0_0_0.ctl

 

Task #3: Create the Repository Database.

Again, I have already covered this task here for 12c Release 5. It’s essentially the same process and is reasonably documented by Oracle here. However, a few things are worth noting.

Note #1: Pick the Correct Template.

The names of the templates are so long now that they extend beyond the width of Template field as shown in the screen shot. That’s fine except you can’t increase the width of the field to see which template is which. I could increase the size of the window, but not the width of the field. Genius!

Click to Open Full Size

 

 

 

 

 

The template selected in the screen shot is the one for a small repository. Clicking any of the radio buttons in the Select column and then clicking the Show Details button will show you more information about that template. How about putting “small”, “medium” or “large” at the beginning of the template name? Thanks Captain Obvious! 😉

Note #2: Do Not Configure Enterprise Manager.

The documentation does mention this, but it’s worth repeating because not following this instruction will lead to bad things happening. When you get to Step #5 in the dbca dialogue, make sure both Enterprise Manager options are UNCHECKED, as shown in the screen shot below.

Click to Open Full Size

 

 

 

 

 

Note #3: Run the shpool SQL Script.

The documentation still includes this warning:

“When you run Oracle Database Configuration Assistant to create the database, on the Database Content screen, by default the Run the following scripts option is selected to run the shpool script. The script sets the shared pool in the database. Make sure you RETAIN THE SELECTION. Do not deselect it.”

There is no Database Content screen as the left panel of the screen shots above clearly show. The script still isn’t called “shpool” either. It’s actually called:

shpool_12.1.0.2.0_Database_SQL_for_EM13_2_0_0_0.sql

Unlike with the 12c Release 5 procedure, there is no option to specify a custom script on the Step 9 screen. This is what the Step 9 screen looks like when using the 13c Release 2 database template:

Click to Open Full Size

 

 

 

 

 

As you can see, no option to specify the shpool SQL script.  Instead, I let the dbca run to completion, then ran the shpool SQL script from a SQL*Plus session once the database had been created:

[oracle@oraemcc templates]$ pwd
/u01/app/oracle/product/12.1.0/dbhome_1/assistants/dbca/templates

[oracle@oraemcc templates]$ . oraenv
ORACLE_SID = [oracle] ? PADMIN
The Oracle base remains unchanged with value /u01/app/oracle

[oracle@oraemcc templates]$ sqlplus / as sysdba
SQL*Plus: Release 12.1.0.2.0 Production on Mon Feb 26 13:59:34 2018
Copyright (c) 1982, 2014, Oracle. All rights reserved.

Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options

SQL> @shpool_12.1.0.2.0_Database_SQL_for_EM13_2_0_0_0.sql
System altered.
System altered.
System altered.

I did a little research on this and it would appear not having the option to specify a SQL script on on the Step 9 screen may have something to do with the template I used. YMMV.

Note #4: The PDB Containing the Management Repository.

Using this template creates a CDB (called PADMIN in my case) and a PDB called EMPDBREPOS. You can call the CDB what you like, but the name of the PDB is fixed. You can tell the PDB contains the Management Repository tablespaces by using these queries:

SQL> show con_name

CON_NAME
------------------------------
CDB$ROOT

SQL> select file_name from dba_data_files;

FILE_NAME
--------------------------------------------------------------------------------
/u02/oradata/PADMIN/datafile/o1_mf_users_f96sgg98_.dbf
/u02/oradata/PADMIN/datafile/o1_mf_undotbs1_f96sghfp_.dbf
/u02/oradata/PADMIN/datafile/o1_mf_system_f96sbzxo_.dbf
/u02/oradata/PADMIN/datafile/o1_mf_sysaux_f96s8dcb_.dbf

SQL> alter session set container=EMPDBREPOS;
Session altered.

SQL> show con_name

CON_NAME
------------------------------
EMPDBREPOS

SQL> select file_name from dba_data_files;

FILE_NAME
--------------------------------------------------------------------------------
/u02/oradata/PADMIN/datafile/o1_mf_users_f96slktd_.dbf
/u02/oradata/PADMIN/datafile/o1_mf_system_f96sfc35_.dbf
/u02/oradata/PADMIN/datafile/o1_mf_sysaux_f96sllyg_.dbf
/u02/oradata/PADMIN/datafile/o1_mf_mgmt_tab_f96smp4m_.dbf
/u02/oradata/PADMIN/datafile/o1_mf_mgmt_ecm_f96sbwrj_.dbf
/u02/oradata/PADMIN/datafile/o1_mf_mgmt_ad4_f96so3by_.dbf

 

Task #4: Install OEM CC 13.2.

There are different hardware and software requirements for installing Release 13.2 depending upon the size and complexity of your infrastructure. We will be performing the simple/small installation. Here are the links to the relevant sections of Oracle’s documentation:

Task #4, Step #1/2: Pre-Installation Check List.

Despite performing a simple/small installation, it makes sense to go through the main items in the pre-installation/prerequisite check list. These are documented in full here, but here are a selection of just 9 to keep you amused:

Check #1: White Space.

Ensure there is no white space in the name of the directory from which you will run the installer:

[oracle@oraemcc ~]$ cd /u01/MEDIA/oem_13cr2p1

[oracle@oraemcc oem_13cr2p1]$ ls -l
-rwxr-xr-x 1 oracle oinstall 2123211088 Feb 24 18:45 em13200p1_linux64-2.zip
-rwxr-xr-x 1 oracle oinstall 741526563 Feb 24 18:45 em13200p1_linux64-3.zip
-rwxr-xr-x 1 oracle oinstall 2084231936 Feb 24 18:46 em13200p1_linux64-4.zip
-rwxr-xr-x 1 oracle oinstall 109191154 Feb 24 18:45 em13200p1_linux64-5.zip
-rwxr-xr-x 1 oracle oinstall 2146696423 Feb 24 18:47 em13200p1_linux64-6.zip
-rwxr-xr-x 1 oracle oinstall 771426157 Feb 24 18:47 em13200p1_linux64-7.zip
-rwxr-xr-x 1 oracle oinstall 554606940 Feb 24 18:47 em13200p1_linux64.bin

Check #2: DISPLAY Variable.

Ensure your DISPLAY environment variable is set accordingly (i.e. to the hostname or IP address of the server from where you’ll run the installation):

[oracle@oraemcc ~]$ export DISPLAY=200.200.10.16:0.0

Check #3: Disable the Optimizer Adaptive Feature.

If you used the database template, then this should have been taken care of already, but let’s check:

SQL> show con_name

CON_NAME
------------------------------
CDB$ROOT

SQL> show parameter optimizer_adaptive_features

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
optimizer_adaptive_features boolean FALSE

SQL> alter session set container=empdbrepos;
Session altered.

SQL> show con_name

CON_NAME
------------------------------
EMPDBREPOS

SQL> show parameter optimizer_adaptive_features

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
optimizer_adaptive_features boolean FALSE

Check #4: Middleware Home Path.

Ensure the full path to the Middleware home directory does not exceed 70 characters:

[oracle@oraemcc middleware]$ pwd
/u01/app/oracle/product/middleware

[oracle@oraemcc middleware]$ pwd | wc -c
35

Check #5: Unique Hostname & Static IP.

Ensure the installation server resolves to a unique hostname and that it uses a static IP address. Note, my DNS server happens to be oraemcc:

[oracle@oraemcc middleware]$ nslookup oraemcc
Server: 200.200.10.16
Address: 200.200.10.16#53

Name: oraemcc.mynet.com
Address: 200.200.10.16

Check #6: Temp Space.

Ensure there is at least 10 GB of temp space. In my case, temporary files will be written to the root (/) file system:

[oracle@oraemcc ~]$ df -h
Filesystem   Size Used Avail  Use% Mounted on
/dev/mapper/vg_oraemcc-lv_root
              33G 6.0G   26G   20%  /
tmpfs        7.9G  72K  7.9G    1% /dev/shm
/dev/xvdb1   477M 123M  325M   28% /boot
/dev/xvdc1    50G  39G   11G   79% /u01
/dev/xvdd1    79G  11G   69G   14% /u02

Check #7: Installation User umask.

Ensure the installation user (oracle) has umask set to 022 in the shell profile file:

[oracle@oraemcc ~]$ umask
0022

Check #8: Unset Environment Variables.

Ensure these environment variables are not set:

[oracle@oraemcc ~]$ echo $CLASSPATH

[oracle@oraemcc ~]$ echo $ORACLE_SID

[oracle@oraemcc ~]$ echo $ORACLE_HOME

Check #9: Network Ports.

In a production environment, I have come across situations where the required ports are being blocked by the corporate firewall. This provides an excellent opportunity to makes friends with the Network Administrator. If you’re really lucky and work in a bureaucratic nightmare, you’ll get to fill out lots and lots of forms to ask really nicely if your ports can be opened up, pretty please. 😀

The port requirements are fully documented here, but essentially boil down to these (YMMV):

Usage Port#
Enterprise Manager Upload HTTP Port 4889
Enterprise Manager Upload HTTP SSL Port 4903
OHS HTTP Port 9788
OHS HTTP SSL Port 9851
Oracle BI Publisher HTTP Port 9701
Oracle BI Publisher HTTP SSL Port 9803
OEM Central Console HTTP Port 7788
OEM Central Console HTTP SSL Port 7802
Node Manager HTTP SSL Port 7403
Managed Server HTTP Port 7202
Managed Server HTTP SSL Port 7301
Management Agent Port 3872
Admin Server HTTP SSL Port 7102

Use the netstat command to figure out if a port is being used. If the command returns no output, then it’s free. Let’s use port #1521 as an example because we know the TNS Listener process is using that port:

[oracle@oraemcc ~]$ netstat -an | grep 1521
tcp 0 0 200.200.10.16:64321 200.200.10.16:1521 ESTABLISHED 
tcp 0 0 :::1521 :::* LISTEN 
tcp 0 0 ::ffff:200.200.10.16:1521 ::ffff:200.200.10.16:64321 ESTABLISHED 
unix 2 [ ACC ] STREAM LISTENING 70836 /var/tmp/.oracle/sEXTPROC1521

Now let’s try port #3872 which will be the Management Agent port (not currently being used):

[oracle@oraemcc ~]$ netstat -an | grep 3872
[oracle@oraemcc ~]$

 

Step #2: Run the Installation Wizard.

Gone are the days when we’d invoke the runInstaller installer. Now it’s called a Wizard. Coloring books and crayons are optional. A couple of things to note here. First, you’ll be executing the .bin file and second, you don’t unzip any of the other files. The ‘wizard’ will do that for us since we can’t be trusted, apparently:

[oracle@oraemcc ~]$ cd /u01/MEDIA/oem_13cr2p1

[oracle@oraemcc oem_13cr2p1]$ ls -l
-rw-r--r-- 1 oracle oinstall 2123211088 Feb 24 18:45 em13200p1_linux64-2.zip
-rw-r--r-- 1 oracle oinstall 741526563 Feb 24 18:45 em13200p1_linux64-3.zip
-rw-r--r-- 1 oracle oinstall 2084231936 Feb 24 18:46 em13200p1_linux64-4.zip
-rw-r--r-- 1 oracle oinstall 109191154 Feb 24 18:45 em13200p1_linux64-5.zip
-rw-r--r-- 1 oracle oinstall 2146696423 Feb 24 18:47 em13200p1_linux64-6.zip
-rw-r--r-- 1 oracle oinstall 771426157 Feb 24 18:47 em13200p1_linux64-7.zip
-rw-r--r-- 1 oracle oinstall 554606940 Feb 24 18:47 em13200p1_linux64.bin

I had to add execute permissions before the .bin file would run:

[oracle@oraemcc oem_13cr2p1]$ chmod 755 *
[oracle@oraemcc oem_13cr2p1]$ ./em13200p1_linux64.bin

Invoking the wizard generates this screen output:

0%...............................................................................100%
Launcher log file is /tmp/OraInstall2018-02-26_07-08-06PM/launcher2018-02-26_07-08-06PM.log.
Starting Oracle Universal Installer

Checking if CPU speed is above 300 MHz. Actual 2665.986 MHz Passed
Checking monitor: must be configured to display at least 256 colors. Actual 16777216 Passed
Checking swap space: must be greater than 512 MB. Actual 16383 MB Passed
Checking if this platform requires a 64-bit JVM. Actual 64 Passed (64-bit not required)

Preparing to launch the Oracle Universal Installer from /tmp/OraInstall2018-02-26_07-08-06PM
====Prereq Config Location main=== 
/tmp/OraInstall2018-02-26_07-08-06PM/stage/prereq
EMGCInstaller args -scratchPath
EMGCInstaller args /tmp/OraInstall2018-02-26_07-08-06PM
EMGCInstaller args -sourceType
EMGCInstaller args network
EMGCInstaller args -timestamp
EMGCInstaller args 2018-02-26_07-08-06PM
EMGCInstaller args -paramFile
EMGCInstaller args /tmp/sfx_ArNIXh/Disk1/install/linux64/oraparam.ini
EMGCInstaller args -nocleanUpOnExit
DiskLoc inside SourceLoc/u01/MEDIA/oem_13cr2p1
EMFileLoc:/tmp/OraInstall2018-02-26_07-08-06PM/oui/em/
ScratchPathValue :/tmp/OraInstall2018-02-26_07-08-06PM

Then what looks suspiciously like an installer screen appears and we’re off to the races. Uncheck the option to receive security updates and click Next:

Click to Open Full Size

 

 

 

 

 

Ignore the patronizing warning message by clicking Yes:

Click to Open Full Size

 

 

 

 

 

Select the Skip radio button and click Next:

Click to Open Full Size

 

 

 

 

 

So here we have a couple of prereq check failures. I left these in to drawn your attention to a ‘gotcha’.

Click to Open Full Size

 

 

 

 

 

We’re using the same server to run both the OMS and the MRDB. Consequently, we needed to install the Database 12c Release 1 database and before that, we installed the Oracle Database 12c Release 1 pre-install package into the OS. That installation is documented here.

Check if it’s installed:

[root@oraemcc ~]# rpm -qa | grep rdbms
oracle-rdbms-server-12cR1-preinstall-1.0-14.el6.x86_64

Installing this package has the effect of setting the local port range to a range the OEM 13.2 installer doesn’t like. Changing it is simple enough:

[root@oraemcc ~]# vi /etc/sysctl.conf
...
# oracle-rdbms-server-12cR1-preinstall setting for net.ipv4.ip_local_port_range is 9000 65500
net.ipv4.ip_local_port_range = 9000 65500

Change the entry to this:

# oracle-rdbms-server-12cR1-preinstall setting for net.ipv4.ip_local_port_range is 9000 65500
#net.ipv4.ip_local_port_range = 9000 65500
net.ipv4.ip_local_port_range = 11000 65000

Activate the change:

[root@oraemcc ~]# sysctl -p
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmall = 4294967296
fs.file-max = 6815744
kernel.sem = 250 32000 100 128
kernel.shmmni = 4096
kernel.shmmax = 4398046511104
kernel.panic_on_oops = 1
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
fs.aio-max-nr = 1048576
net.ipv4.ip_local_port_range = 11000 65000

The softnofiles issue is slightly more devious. The soft and hard limits for the number of files a process can have open is (usually) set in the file, /etc/security/limits.conf:

[root@oraemcc ~]# vi /etc/security/limits.conf

...
oracle   soft   nofile  8192
oracle   hard   nofile  65536
...

However, the error is reporting the soft limit is currently set to 16384 which, according to the value in /etc/security/limits.conf, is NOT the current soft limit. Trust me, setting either the hard or soft limit in this file will have no effect on the error the OEM 13.2 installer is reporting. So what’s going on?

Installing the pre-install RDBMS package creates a configuration file which overrides the limits.conf file:

[root@oraemcc ~]# cd /etc/security/limits.d
[root@oraemcc limits.d]# ls -l

-rw-r--r--. 1 root root  191 Aug 18  2015 90-nproc.conf
-rw-r--r--  1 root root 1093 Feb 26 21:52 oracle-rdbms-server-12cR1-preinstall.conf

[root@oraemcc limits.d]# view oracle-rdbms-server-12cR1-preinstall.conf

# oracle-rdbms-server-12cR1-preinstall setting for nofile soft limit is 1024
oracle   soft   nofile    1024
# oracle-rdbms-server-12cR1-preinstall setting for nofile hard limit is 65536
oracle   hard   nofile    16384
...

This is where the OEM 13.2 installer is getting its information from and is actually using the hard limit, not the soft limit. Editing this file will fix the problem, but you will need to reboot afterwards, re-start the MRDB   and start the OEM 13.2 installation again.

[root@oraemcc limits.d]# vi oracle-rdbms-server-12cR1-preinstall.conf

# oracle-rdbms-server-12cR1-preinstall setting for nofile soft limit is 1024
oracle   soft   nofile    30000
# oracle-rdbms-server-12cR1-preinstall setting for nofile hard limit is 65536
oracle   hard   nofile    65536

Notice it’s the hard limit value the OEM 13.2 installer references. With all the prereq checks successfully completed, click Next:

Click to Open Full Size

 

 

 

 

 

Select the Simple option of Create a new Enterprise Manager system, then click Next:

Click to Open Full Size

 

 

 

 

 

Use these values to populate this next screen, then click Next:

Field Value
Middleware Home Location /u01/app/oracle/product/middleware
Agent Base Directory /u01/app/oracle/product/agent
Home Name oraemcc.mynet.com

Click to Open Full Size

 

 

 

 

 

 

Dream up a suitable password for all the accounts mentioned in the next screen. Write it down now. You will need it shortly. Also, use these values to populate this next screen, then click Next:

Field Value
Database Host Name oraemcc.mynet.com
Port 1521
Service/SID emdbrepos.mynet.com
SYS Password The SYS password you used to create the MRDB

Click to Open Full Size

 

 

 

 

 

Now we hit another little gotcha. Clicking Next on the previous screen attempted to make a connection to the PDB containing the Management Repository. However, when we rebooted earlier, the CDB probably came back up, but the PDB probably did not. If that was the case, you see this rather misleading error message:

Click to Open Full SIze

 

 

 

 

 

Let’s login to the CDB and find out what’s going on:

SQL> select name, open_mode from v$pdbs;

NAME                           OPEN_MODE
------------------------------ ----------
PDB$SEED                       READ ONLY
EMPDBREPOS                     MOUNTED

SQL> alter pluggable database EMPDBREPOS open;
Pluggable database altered.

SQL> select name, open_mode from v$pdbs;

NAME                           OPEN_MODE
------------------------------ ----------
PDB$SEED                       READ ONLY
EMPDBREPOS                     READ WRITE

Click the OK button on the error window, then click Next again. That will land you on this next screen. I’m not interested in Oracle BI Publisher, so I unchecked those options and left the Software Library Location at its default value (/u01/app/oracle/product/swlib):

Click to Open Full Size

 

 

 

 

 

Click Next and we end up on a summary screen. Here’s the top half:

Click to Open Full Size

 

 

 

 

 

Here’s the bottom half showing all the default ports:

Click to Open Full Size

 

 

 

 

 

Click the Install button and then wait. Your next task will be to run a root script. It took close to 2 hours to reach that stage on my system. YMMV:

Click to Open Full Size

 

 

 

 

 

Time passes:

Click to Open Full Size

 

 

 

 

 

Eventually you’ll see this window appear:

Click to Open Full Size

 

 

 

 

 

In a separate root shell session, run the allroot.sh script then click the OK button:

[root@oraemcc middleware]# ./allroot.sh

Starting to execute allroot.sh .........
Starting to execute /u01/app/oracle/product/middleware/root.sh ......
/etc exist
/u01/app/oracle/product/middleware
Finished product-specific root actions.
/etc exist
Finished execution of  /u01/app/oracle/product/middleware/root.sh ......

Starting to execute /u01/app/oracle/product/agent/agent_13.2.0.0.0/root.sh ......
Finished product-specific root actions.
/etc exist

Finished execution of  /u01/app/oracle/product/agent/agent_13.2.0.0.0/root.sh ......

Finally, you’ll see an installation summary screen. Click Close and you’re done!

Click to Open Full Size

 

 

 

 

 

 

All that remains is to scoot over to your OEM CC 13.2 URL and login as SYSMAN:

Click to Open Full Size

 

 

 

 

 

 

If you have any comments or questions about this post, please use the Contact form here.

Synology NAS

 

Some tips and tricks to help get the most out of the Synology DS413 NAS.

 

 

Use the following links to jump to the item you need.

Index:

 

Accessing the root Account.

Sometimes you need to do things as the root user. To gain access to the root account is a multi-step process. Here’s what I do:

Fire up Putty and login as the admin user. The password will be the same as the admin user you setup when you initially configured your NAS. In order to get a Putty session established, you will need to ensure SSH is enabled. To do that, open up Control Panel, click Terminal & SNMP and check the Enable SSH service box.

Once you’re logged in as admin, enter this command:

admin@YOURNAS:/$ sudo su -
Password: <your admin password>
root@YOURNAS:~#

 

Adding Entries to /etc/hosts.

One thing you’ll need root access for is adding entries to the /etc/hosts file. This is useful if you want to take advantage of hostname resolution when setting up NFS client access on your NAS. You can use the IP address of NFS clients, so it’s not strictly necessary to add entries to /etc/hosts. Note, your custom entries to /etc/hosts do survive a DiskStation Manager (DSM) upgrade. You can use vi to edit /etc/hosts.

 

Setting Up NFS Access.

To conserve space on your Linux servers, it’s often a good idea to store Oracle media on an NFS drive rather than directly on the server itself. Setting up NFS access is a two stage process. Stage one is to setup NFS permissions via the DSM. Stage two is to configure an NFS share on your Linux server.

Stage One:

After creating a volume using the DSM, you’ll need to configure NFS client access to that volume.

In DSM open up Control Panel. Click on Shared Folder. Right click the relevant volume, then click Edit. Once you’re in the Edit window, click on the NFS Permissions tab. Click the Create button. This will enable you to create an NFS rule. I use these entries:

 

Field Value
Hostname or IP*: NFS client IP or hostname (from /etc/hosts)
Privilege: Read/Write
Squash: No mapping (allows file ownership changes)
Enable asynchronous Check
Allow connections from non-privileged ports Leave Unchecked
Allow users to access mounted subfolders Check

Click OK and the NAS will process your changes. That’s the NAS side of things completed.

Stage Two:

The second stage is to add some configuration to the Linux server.

Add a mount point directory in the root file system. Let’s call it nas (how original!):

[root@orasvr01 ~]$ cd /
[root@orasvr01 /]$ mkdir nas

Make the mount point directory accessible to everyone:

[root@orasvr01 ~]# chmod 777 /nas

Edit the /etc/fstab to include this entry:

YOURNAS:/volume1/NASNFS   /nas   nfs     rw,intr,rsize=16384,wsize=16384,timeo=10 0 0

where:

Value Meaning
YOURNAS Name of your NAS device
/volume1/NASNFS Name of the volume you created using DSM
/nas Name of the mount point directory
nfs Type of file system

Save /etc/fstab, then mount the NFS share using this command:

[root@orasvr01 ~]# mount –all

This will scan the /etc/fstab and mount any currently unmounted file systems.

Verify your NFS share is mounted using this command:

[root@orasvr01 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_orasvr01-lv_root
                       31G   12G   18G  40% /
tmpfs                 3.9G   72K  3.9G   1% /dev/shm
/dev/xvdb1            477M  123M  325M  28% /boot
/dev/xvdc1             30G   23G  6.8G  78% /u01
/dev/xvdd1             30G   45M   30G   1% /u02
YOURNAS:/volume1/NASNFS
                      3.6T  1.1T  2.6T  30% /nas

There you have it! Your Linux server now has additional file system storage coming from your NAS.