Monday, August 30, 2021

Boot OL8 in single user mode to reset root password.

2:46 PM Posted by Dilli Raj Maharjan No comments

Booting OL8 in single user mode to reset root password.

Press e to edit the boot menu.



Once boot options is displayed, add rd.break at the end of the linux.. . Below is the screenshot of the rd.break at the end. Once rd.break is added Press CTRL + x to boot system in single user mode.



Once booted into single user mode, it looks something like below.



Remount /sysroot with rw and chroot to /sysroot.



Now change password with passwd command.



Once password is changed. Type exit get out of chroot. Type reboot to restart normal.



Sunday, August 29, 2021

Applying PSU Oracle database 19c

5:08 PM Posted by Dilli Raj Maharjan , No comments

    



    Since Oracle database version 12.2.0.1, Oracle has changed patching model from Patch Set Updates(PS) and Proactive Bundle Patches(BP) to Release Updates(RU) and Release Update Revisions(RUR). Release updates changes the second number from the database version. Following are the steps by step guide to apply RU and upgrading database from 19.3.0.0.0 to 19.7.0.0.0.

       Go to MOS document below and download Oracle RU and opatch utility. It is always good idea to download latest version of the opatch from MOS site. Search for patch number 6880880 to download the latest version of opatch. If you are not able to download the latest version of opatch then go to Patch README.html file and make sure the version of patch you have match the least required version of opatch. 

Primary Note for Database Proactive Patch Program (Doc ID 888.1)

Once you have downloaded the patches. Extract them
unzip p30869156_190000_Linux-x86-64.zip
unzip p6880880_190000_LINUX.zip

List of the patches downloaded from MOS site.

Extracted DB and OJVM combo patch.

Extracted Opatch utilities.


Tree view of the README.html file. Make sure you read the README.html file in details for pre-requisites and the steps to apply patches.



Check existing opatch version in the database server.
cd $ORACLE_HOME/OPatch
./opatch version



Copied latest version of opatch and checked version. But there were errors.
cd $ORACLE_HOME
mv OPatch/ OPatch.001
mv /mnt/Ora19c_Patches/OPatch .


Checked java version and it was 1.8.0_302.



 While searching in an Internet for the solution I finally found the way to fix it by adding option jdk at the end of the command as -jdk $ORACLE_HOME/jdk.
cd $ORACLE_HOME/OPatch
./opatch version -jdk $ORACLE_HOME/jdk



Shutdown the database prior to applying patch.



Change directory to DB patch. In my case directory 30869156 contains the DB Patches.
cd 30869156


Export PATH environment variables to include the location of the latest version of opatch. Validate the opatch location using which linux command.
export PATH=$ORACLE_HOME/OPatch:$PATH
which opatch



Check if there is any conflicting interim patches installed in the database with command below.
opatch prereq CheckConflictAgainstOHWithDetail -ph ./ -jdk $ORACLE_HOME/jdk



Once conflict check is passed apply PSU with command below.
opatch apply -jdk $ORACLE_HOME/jdk



Login to database and start database and pdb(s) if applicable.
sqlplus /nolog
conn / as sysdba
startup
alter pluggable database all open;
quit



Applied datapatch with verbose option but noticed error ORA-20001.


Error ORA-20001: Latest xml inventory is not loaded into table



Search on an Internet and modified hidden parameter _disable_directory_link_check to TRUE. It did not worked for me :(
alter system set "_disable_directory_link_check"=TRUE scope=spfile;
shutdown immediate
startup
alter pluggable database all open;



Next solution tried is to created directory OPATCH_LOG_DIR and OPATCH_SCRIPT_DIR. It did not worked for me :( :(
create or replace directory OPATCH_LOG_DIR as
'/u01/app/oracle/product/19.3.0/dbhome_1/QOpatch';
create or replace directory OPATCH_SCRIPT_DIR as
'/u01/app/oracle/product/19.3.0/dbhome_1/QOpatch';



While reading site https://mikedietrichde.com/2020/07/31/upgrade-fails-with-ora-20001-during-datapatch-run/ I came to know about the inventory table OPATCH_XML_INV. I checked the table and noticed the error was due to the opatch.
select * from OPATCH_XML_INV;



I opened the file qopiprep.bat file in the location mentioned above and added option jdk in the command opatch.
cd $ORACLE_HOME/QOpatch
vi qopiprep.bat




I accessed the inventory table once again and it worked.
select * from OPATCH_XML_INV;



Applied datapatch  with verbose option and it succeed.
cd $ORACLE_HOME/OPatch
./datapatch -verbose


Once datapatch is completed shutdown database to apply OJVM Patch.
conn / as sysdba
shutdown immediate
exit



Change directory to OJVM patch. Check if any conflicting patch is installed for OJVM patch.


Apply OJVM Patch



Start database in upgrade mode


Apply datapatch with verbose option



Shutdown and start database in normal mode.



Query DBA Patch registry.



Execute opatch lspatches command to list the patches applied.



Saturday, August 29, 2020

Verify readiness of Database switchover using DG Validate command

5:55 PM Posted by Dilli Raj Maharjan , , , 8 comments
    
    Oracle Dataguard ensures high availability, data protection, and disaster recovery for Oracle database. Data Guard provides a comprehensive set of services for DR solution. The services can be used to create, maintain, manage, monitor one or more standby databases, and validate the database for the readiness of role switch. Data Guard can switch any standby database manually or automatically in case the production database becomes unavailable due to a planned or unplanned outage. Data Guard minimizes the downtime associated with the outage.

    Oracle Data Guard is included with the Enterprise Edition and Personal Edition of the Oracle database software. We can manage an Oracle Data Guard configuration by using either SQL*Plus, the Oracle Data Guard broker's command-line interface (DGMGRL), or a compatible version of Oracle Enterprise Manager Cloud Control (Cloud Control).

    Validate command has a set of options used to check a comprehensive set of checks prior to role change. This command is used to verify the readiness of switchover and to validate the standby database is ready to failover and switchover anytime. Validate command was first introduced in Oracle database 12.1.0.0. New commands like validate database spfile, validate network configuration, validate static connect identifier was introduced in Oracle Database Release 18c, Version 18.1. For more details click here.

Environment:

Docker Image: dillimaharjan/oracle19c
Docker Container: ora19cp, ora19cs
Oracle Database version: 19.3.0.0
Oracle Database Edition: Oracle Database 19c Enterprise Edition
Container database: False

In my case, there are docker images stored in my repo for different versions of the database. I have executed docker image command to list all available images. oracle19c is a docker image with fresh Oracle 19.3.0.0 installed and ready to use.

docker image ls | sort




Start docker container using the docker run command. The following are a brief description of the option being used.
-i                     : Interactive mode
-t                     : Allocate a pseudo-TTY
-d                    : Detach once container is started
--privileged    :  Give an extended privileges to the container
-v             : Bind mount a volume. OS directory will be mounted to the specified mount point in the container.
-p                    : Publish a container port to the host
--hostname     : Host name of the container.
--name            : Name of the container.

docker run -i -t -d --privileged -v /Users/dilli/Docker/shared:/shared -p 15219:1521 --hostname ora19c_primary --name ora19cp dillimaharjan/oracle19c
docker run -i -t -d --privileged -v /Users/dilli/Docker/shared:/shared -p 15220:1521 --hostname ora19c_standby --name ora19cs dillimaharjan/oracle19c



List running docker containers with docker ps command.
docker ps



Run bash in a running container to get bash terminal of the container.
docker exec -it ora19cp /bin/bash
docker exec -it ora19cs /bin/bash




Physical standby and the DG have been configured for those docker containers. If you need any help to configure physical standby or DG. Please refer to the links below.


Once DG configuration is completed, the configuration looks like below. The command show configuration will list the primary and standby database along with the FSFO and configuration status.

show configuration.





1. Validate Database

The validate database command is used for a comprehensive set of database checks prior to a role change. Role change from Primary to Standby and from Standby to Primary. This command is used to verify the readiness prior to the switchover.

Syntax:
VALIDATE DATABASE [VERBOSE] <database-name>;

Validating primary database. 

The Ready for Switchover is Yes, which means the database has no issue and ready to switchover anytime.

validate database accdb;



Verbose option will reveal more details about the database. 
validate database verbose accdb;



Validating standby database.

The Ready for Switchover and the Ready for Failover both have value Yes, which means the database is ready for switchover and failover.

validate database accstdb;



validate database verbose accstdb;





2. Validate database datafile

The validate database datafile validates data files across the primary and standby database. Option ALL can be defined to validate all the datafiles on either of the database or all databases. OUTPUT=<output_file_name> is mandatory and command dumps output to the file. Output should not contain any directory. The output file will be store in the trace directory.


Syntax:
VALIDATE DATABASE [database-name | ALL] DATAFILE [datafile-name | datafile-number | ALL] OUTPUT="<output file name>";

List the available data files in the database.

select file# from v$database;



If you specify the directory in the output file it will return error: DGM-17138: OUTPUT should not contain a directory. The output file will be saved in the trace file location.


Validate datafile # 1 in database accdb and save all output to accdb_dbfile_1.txt file in the trace location.
Additionally, validate all datafiles in accdb database, validate datafile # 1 in accstdb, and validate all datafiles in all databases.

validate database accdb datafile 1 output='accdb_dbfile_1.txt'
validate database accdb datafile all output='accdb_all_dbfiles.txt'
validate database accstdb datafile 1 output='accstdb_dbfile_1.txt';
validate database all datafile all output='all_datafiles.txt'




Sample content of the output file.

Client is connected to database: accdb. Role: primary database.

Remote database accstdb.remote db role: physical standby


Slave Id  0

Summary:

*******************************************************************************

                       TOTAL: total no. of blocks found

                         |

   +--------+------------+-------+---------+---------+

   |        |                    |         |         |

   |      DIFFV:             LOST_WRITE    |       CORR: corrupted blocks

 SAMEV     diff ver              |       SKIPPED:

   |       block pairs        +--+--+      direct load, empty blocks,

+--+--+--+                    |     |      RMAN optimized blocks,

|  |  |  |                    |     |      flashback optimized blocks

|  |  | SAMEV&C:              |     |

|  |  |  same ver &           |   LWLOC: lost writes at local db

|  |  |  same checksum &     LWRMT: lost writes at remote db

|  |  |  same contents

|  |  |

|  | SAMEV_NO_CHKSUM: same ver & same contents but diff checksum

|  |                  (checksum can be diff but identical contents)

|  |

| DIFFPAIR: same ver but differrent contents (data inconsistency)

|

ENCERR: undecided block pairs due to encryption related issue

        (e.g. when Wallet is not open)



ID TOTAL   CORR SKIPPED DIFFV   SAMEV   SAMEV&C ENCERR  LWLOC  LWRMT DIFFPAIR

00 0039190 0000 0039189 0000001 0000000 0000000 0000000 000000 000000 0000000

02 0042599 0000 0000000 0000005 0042594 0042566 0000000 000000 000000 0000000

06 0126447 0000 0027918 0000286 0098243 0098224 0000000 000000 000000 0000000

14 0000001 0000 0000000 0000000 0000001 0000001 0000000 000000 000000 0000000


3. Validate database spfile

The validate spfile command validates the difference in spfile between the primary and the standby database. Executing this command prior to switchover lists out all the parameter differences. Setting parameters of primary and standby exactly the same may prevent performance issues after switchover. If all the parameter values are the same then "No parameter difference found." message will be displayed.

Syntax:
VALIDATE DATABASE [verbose] {standby database-name} spfile;

Note: You should log in as user sys to execute following command
If you log in with os authentication in dgmgrl then the following error will be displayed
Command requires a connection that uses database or external credentials.



Note: Validate spfile command cannot be executed for the Primary database. The following error will be encountered if you execute validate spfile for the primary database.
This command cannot be used for the primary database.



Validate database <standby database> spfile;



Verbose option will display matched and unmatched parameter values.



4. Validate FAR_SYNC


The validate far_sync validates far sync instance. It performs a set of checks for a far sync instance.


Syntax:
VALIDATE FAR_SYNC VERBOSE] far_sync_instance_name [WHEN PRIMARY IS <primary db_name>]


5. Validate Network configuration.

The validate network configuration command checks the network configuration between members. It performs network connectivity checks and reports for any issues.


Syntax:
VALIDATE NETWORK CONFIGURATION FOR <ALL | MEMBER NAME>;


In my case the hostname of standby is not resolvable from Primary and hostname of Primary is not resolvable from Standby because these settings are missing on my /etc/hosts file. So I have encountered the following error.


Once my error is resolved the validate command passed.

validate network configuration for all;
validate network configuration for accdb;
validate network configuration for accstdb;




6. Validate static connect identifier.

The validate static connect identifier validates the static connect identifier of a database. Static connect identifiers are required to start the database during the switchover.


Syntax:
VALIDATE STATIC CONNECT IDENTIFIER FOR <ALL | DATABASE NAME>;


In my case I have not set <server_name>_DGMGRL static server name in the listener configuration that is why I am getting error ORA-12514: TNS:listener does not currently know of service requested in connect descriptor.





Once I added static service name <service>_DGMGRL the validate command passed.

validate static connect identifier for all;
validate static connect identifier for accdb;
validate static connect identifier for accstdb;







Switchover

Once all my validate commands passed, I have executed switchover and it succeed without any issue.


Now accstdb is my primary database and accdb is standby database.