CRS falla al arrancar con el error CRS-6706: Oracle Clusterware Release patch level

Hoy vamos a ver una sencilla entrada que puede ocurrirnos al parchear el ASM

Los recursos de crs no levantan y recibimos el eror

[testserver]$ sudo $ORACLE_HOME/bin/crsctl start has
CRS-6706: Oracle Clusterware Release patch level ('3930441427') does not match Software patch level ('526767740'). Oracle Clusterware cannot be started.
CRS-4000: Command Start failed, or completed with errors.

En este caso oracle nos recomienda que desbloqueemos el crs y repitamos los pasos .
Aunque en la nota de oracle nos indiquen ejecutar

# /crs/install/roothas.sh -patch

Este es un comando muy vago, por lo que, mi recomendacion es, sabiendo donde ha fallado, ejecutar el pre o ppost pach

[testserver]$ sudo $ORACLE_HOME/crs/install/roothas.sh -prepatch
Using configuration parameter file: /u01/app/oracle/product/19c/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/oracle/crsdata/icdbstst01/crsconfig/hapatch_2023-08-09_10-55-17AM.log
2023/08/09 10:55:29 CLSRSC-347: Successfully unlock /u01/app/oracle/product/19c/grid
2023/08/09 10:55:29 CLSRSC-671: Pre-patch steps for patching GI home successfully completed.

[testserver]$ sudo $ORACLE_HOME/crs/install/roothas.sh -postpatch
Using configuration parameter file: /u01/app/oracle/product/19c/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/oracle/crsdata/icdbstst01/crsconfig/hapatch_2023-08-09_10-55-36AM.log
2023/08/09 10:55:40 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.service'
2023/08/09 10:58:01 CLSRSC-672: Post-patch steps for patching GI home successfully completed.

[testserver]$ sudo $ORACLE_HOME/bin/crsctl start has
CRS-4640: Oracle High Availability Services is already active
CRS-4000: Command Start failed, or completed with errors.

Como veis, nada de que asustarse con los comandos indicados

Mas information en soporte con la nota

  • CRS-6706: Oracle Clusterware Release patch level (‘nnn’) does not match Software patch level (‘mmm’) (Doc ID 1639285.1)

Corrupcion de bloques no detectados por validate

Hoy vamos a ver una entada que nos puede traer un poco de cabeza.

Supongamos tenemos la tipica consulta que nos devuelve un error de corrupcion de bloque ORA-01578: ORACLE data block corrupted

SQL> select something from sometable  where file_name='whatever';
ERROR at line 1:
ORA-01578: ORACLE data block corrupted (file # 377, block # 2818432)
ORA-01110: data file 377:
'+DATA/TESTDB/DATAFILE/TESTDB.20221119.110001.377.dbf' 

Ante este error, nuestros pasos suelen ser claros.

  • Buscamos los bloques corruptos con rman validate o en su defecto dbverify dbv
  • Comprobamos la vista V$DATABASE_BLOCK_CORRUPTION;
  • Recuperamos de backup datafile & block recover datafile 377block 2818432;

Pero , que ocurre si tras ejecutar el validate o el dbv ?

Si miramos el alert.log durane la validacion, veremos lineas con el contenido

TESTDB(3):Completely zero block found during validation

Esto nos indica que es un error en un bloque que contiene solo ceros.
Oracle por diseño no escribe bloques con todo ceros, por lo que este error es heredado del sistema operativo o el sistema de almacenamiento.

Para solucionar este problema deberemos de recuperar ese datafile desde un backup con cualquiera de los metodos soportados

Mas informacion como siempre en Oracle Support

  • Physical Corrupted Blocks consisting of all Zeroes indicate a problem with OS, HW or Storage (Doc ID 1545366.1)

ORA-600 en la ejecucion de DBMS_STATS.GATHER_FIXED_OBJECTS_STATS;

Hoy vamos a ver una sencilla entrada sobre un paso basico en un upgrade de base de datos.

Antes de llevar a cabo un proceso de migracion debemos de asegurarnos que todas las estadisticas de la base de datos estan recientemente recopiladas ( menos de fos dias),

Puede darse el caso de que ejecutando el paquete de obtencion de las estadisticas de los Fixed Objects recibamos el error

 sqlplus "/as sysdba"
execute DBMS_STATS.GATHER_FIXED_OBJECTS_STATS;
BEGIN DBMS_STATS.GATHER_FIXED_OBJECTS_STATS; END;
*
ERROR at line 1:
ORA-20011: Approximate NDV failed: ORA-00600: internal error code, arguments:
[1350], [3], [23], [60], [AMERICAN], [0], [], [], [], [], [], []
ORA-06512: at "SYS.DBMS_STATS", line 37237
ORA-06512: at line 1

Esto puede darse cuando tenemos la obtencion de estadisticas en auto_sample.
Para solucionarlo, nada mas facil que mover el porcentaje de estimacion a NULL


exec dbms_stats.set_global_prefs(pname=>’ESTIMATE_PERCENT’,pvalue=>’NULL’);


SQL>  select dbms_stats.get_prefs('ESTIMATE_PERCENT') from dual;
DBMS_STATS.GET_PREFS('ESTIMATE_PERCENT')
--------------------------------------------------------------------------------
DBMS_STATS.AUTO_SAMPLE_SIZE

SQL> exec dbms_stats.set_global_prefs(pname=>'ESTIMATE_PERCENT',pvalue=>'NULL');
PL/SQL procedure successfully completed.
SQL> select dbms_stats.get_prefs('estimate_percent') from dual;
DBMS_STATS.GET_PREFS('ESTIMATE_PERCENT')
--------------------------------------------------------------------------------
NULL

Con este sencillo paso podremos ejecutar nuestro paquete de obtencion de estadisticas en los objetos fijos sin eosos errores ORA

Comandos basicos del Trace File Analyzer

Hoy vamos a ver los comandos y acciones basicas del tafcl
El tafcl es el comando de ejecucion del Trace File Analyzer (TFA), este pequeño script esta incluido dentro de la suite Autonomous Health Framework (AHF) , y sera o primero que nos solicite Oracle support una vez abramos un SR

Instalacion

Nos descargaremos el paquete de la AHF de la nota de soporte Autonomous Health Framework (AHF) – Including TFA and ORAchk/EXAchk (Doc ID 2550798.1 Y lo ejecutaremos como superusuario el comando ahf_setup.
Este comando tiene varias opciones

   Usage for ./ahf_setup

   ./ahf_setup [[-ahf_loc ][-data_dir ][-nodes ][-extract[orachk|exachk|-notfasetup] [-local][-silent][-tmp_loc ][-force][-debug [-level <1-6>]]]

        -ahf_loc          -    Install into the directory supplied. (Default /opt/oracle.ahf)
        -data_dir         -    AHF Data Directory where all the collections, metadata, etc. will be stored
        -nodes            -    Comma separated Remote Node List
        -extract          -    Extract only files from Installer. (Default for non-root users)
        -nosymlink        -    Do not create symlinks on Exadata DOM0. (Default for non-root users)
        -notfasetup       -    Do not Configure TFA when used with -extract
        -local            -    Only install on the local node
        -silent           -    Do not ask any install questions
        -tmp_loc          -    Temporary location directory for AHF to extract the install archive to (must exist) (Default /tmp)
        -perlhome         -    Custom location of perl binaries
        -force            -    Force AHF Install
        -debug            -    Debug AHF Install Script
        -level            -    AHF Instal Debug Level 1-6 (Default 4 with option -debug)
                               [FATAL - 1, ERROR - 2, WARNING - 3, INFO - 4, DEBUG - 5, TRACE - 6]

Si la instalacionn se hizo de manera estandard, el mismo TFA detectara la instalacion anterior y se actualizara.
Veamos un ejemplo de actualizacion.

[oracle@rac1 tmp]$ sudo ./ahf_setup 

AHF Installer for Platform Linux Architecture x86_64

AHF Installation Log : /tmp/ahf_install_211000_21804_2021_04_01-09_22_26.log

Starting Autonomous Health Framework (AHF) Installation

AHF Version: 21.1.0 Build Date: 202104131158

AHF is already installed at /opt/oracle.ahf

Installed AHF Version: 20.2.3 Build Date: 202010132201

Do you want to upgrade AHF [Y]|N : Y

Upgrading /opt/oracle.ahf

Shutting down AHF Services
Shutting down TFA
Removed symlink /etc/systemd/system/multi-user.target.wants/oracle-tfa.service.
Removed symlink /etc/systemd/system/graphical.target.wants/oracle-tfa.service.
. . . . .
. . .
Successfully shutdown TFA..


Starting AHF Services
Starting TFA..
Created symlink from /etc/systemd/system/multi-user.target.wants/oracle-tfa.service to /etc/systemd/system/oracle-tfa.service.
Created symlink from /etc/systemd/system/graphical.target.wants/oracle-tfa.service to /etc/systemd/system/oracle-tfa.service.
Waiting up to 100 seconds for TFA to be started..
. . . . .
. . . . .
. . . . .
Successfully started TFA Process..
. . . . .
TFA Started and listening for commands
No new directories were added to TFA


INFO: Starting orachk scheduler in background. Details for the process can be found at /opt/app/oracle/oracle.ahf/data/rac1/diag/orachk/compliance_start_230421_093124.log


AHF is successfully upgraded to latest version

.------------------------------------------------------------------.
| Host       | TFA Version | TFA Build ID         | Upgrade Status |
+------------+-------------+----------------------+----------------+
| rac1 |  21.1.0.0.0 | 21100020210413115827 | UPGRADED       |
| rac2 |  20.2.3.0.0 | 20230020201013220107 | NOT UPGRADED   |
'------------+-------------+----------------------+----------------'

Moving /tmp/ahf_install_211000_21804_2021_04_01-09_22_26.log to /opt/app/oracle/oracle.ahf/data/rac1/diag/ahf/

Usos comunes

Obtener informacion de las trazas en un determninado tiempo

Si quremos recabar informacion del sistema entre unas determinadas fechas usaremos los fags -from  -to
./tfactl diagcollect -from "2021-04-07 08:00:00" -to "2021-04-07 10:00:00"

Obtener informacion ded un determinado error

Si quremos obtener informacion del sistema sobre un determinado error, usaremos el flag srdc

./tfactl diagcollect -srdc ORA-07445

Gestion del repositorio

Ver la configuration

 
tfactl> print config
.---------------------------------------------------------------------------------------------------------------------.
|                                                      wiracdev02                                                     |
+--------------------------------------------------------------------------------------------------------+------------+
| Configuration Parameter                                                                                | Value      |
+--------------------------------------------------------------------------------------------------------+------------+
| TFA Version ( tfaversion )                                                                             | 21.1.0.0.0 |
| Java Version ( javaVersion )                                                                           | 1.8        |
| Public IP Network ( publicIp )                                                                         | true       |
| Repository current size (MB) ( currentsizemegabytes )                                                  | 293        |
| Repository maximum size (MB) ( maxsizemegabytes )                                                      | 10240      |
| Cluster Event Monitor ( clustereventmonitor )                                                          | ON         |
| scandiskmon                                                                                            | OFF        |
| scanacfslog                                                                                            | OFF        |
| File Data Collection ( inventory )                                                                     | ON         |
| Automatic Purging ( autoPurge )                                                                        | ON         |
| Internal Search String ( internalSearchString )                                                        | ON         |
| ISA Data Gathering ( collection.isa )                                                                  | ON         |
| Trim Files ( trimfiles )                                                                               | ON         |
| collectTrm                                                                                             | OFF        |
| chmdataapi                                                                                             | ON         |
| chanotification ( chanotification )                                                                    | ON         |
| Managelogs Auto Purge ( manageLogsAutoPurge )                                                          | OFF        |
| scanacfseventlog                                                                                       | OFF        |
| Alert Log Scan ( rtscan )                                                                              | ON         |
| debugips                                                                                               | OFF        |
| generateZipMetadataJson                                                                                | ON         |
| collectAllDirsByFile                                                                                   | ON         |
| scanvarlog                                                                                             | OFF        |
| Auto Diagcollection ( autodiagcollect )                                                                | ON         |
| Public IP Network ( publicIp )                                                                         | ON         |
| Flood Control ( floodcontrol )                                                                         | ON         |
| Generation of Mini Collections ( minicollection )                                                      | ON         |
| odscan                                                                                                 | ON         |
| Disk Usage Monitor ( diskUsageMon )                                                                    | ON         |
| Discovery ( discovery )                                                                                | ON         |
| analyze                                                                                                | OFF        |
| indexInventory                                                                                         | ON         |
| Generation of Telemetry Data ( telemetry )                                                             | OFF        |
| chaautocollect                                                                                         | ON         |
| Granular Tracing ( granulartracing )                                                                   | OFF        |
| minPossibleSpaceForPurge                                                                               | 1024       |
| disk.threshold                                                                                         | 90         |
| mem.swapfree                                                                                           | 5120       |
| mem.util.samples                                                                                       | 4          |
| inventoryThreadPoolSize                                                                                | 1          |
| mem.swaptotal.samples                                                                                  | 2          |
| maxFileAgeToPurge                                                                                      | 1440       |
| mem.free                                                                                               | 20480      |
| actionrestartlimit                                                                                     | 30         |
| Minimum Free Space to enable Alert Log Scan (MB) ( minSpaceForRTScan )                                 | 500        |
| cpu.io.samples                                                                                         | 30         |
| mem.util                                                                                               | 80         |
| Maximum single Zip File Size (MB) ( maxZipSize )                                                       | 2048       |
| Time interval between consecutive Disk Usage Snapshot(minutes) ( diskUsageMonInterval )                | 60         |
| TFA ISA Purge Thread Delay (minutes) ( tfaDbUtlPurgeThreadDelay )                                      | 60         |
| firstDiscovery                                                                                         | 0          |
| TFA IPS Pool Size ( tfaIpsPoolSize )                                                                   | 5          |
| Maximum File Collection Size (MB) ( maxFileCollectionSize )                                            | 5120       |
| Time interval between consecutive Managelogs Auto Purge(minutes) ( manageLogsAutoPurgeInterval )       | 60         |
| arc.backupmissing.samples                                                                              | 2          |
| cpu.util.samples                                                                                       | 2          |
| cpu.usr.samples                                                                                        | 2          |
| cpu.sys                                                                                                | 50         |
| Flood Control Limit Count ( fc.limit )                                                                 | 3          |
| Flood Control Pause Time (minutes) ( fc.pauseTime )                                                    | 120        |
| Maximum Number of TFA Logs ( maxLogCount )                                                             | 10         |
| DB Backup Delay Hours ( dbbackupdelayhours )                                                           | 27         |
| cdb.backup.samples                                                                                     | 1          |
| arc.backupstatus                                                                                       | 1          |
| purgeFrequency                                                                                         | 4          |
| TFA ISA Purge Age (seconds) ( tfaDbUtlPurgeAge )                                                       | 2592000    |
| Maximum Collection Size of Core Files (MB) ( maxCoreCollectionSize )                                   | 200        |
| cpu.util                                                                                               | 80         |
| mem.swapfree.samples                                                                                   | 2          |
| cdb.backupstatus                                                                                       | 1          |
| mem.swaputl.samples                                                                                    | 2          |
| arc.backup.samples                                                                                     | 3          |
| unreachablenodeTimeOut                                                                                 | 3600       |
| Flood Control Limit Time (minutes) ( fc.limitTime )                                                    | 60         |
| mem.swaputl                                                                                            | 10         |
| mem.free.samples                                                                                       | 2          |
| Maximum Size of Core File (MB) ( maxCoreFileSize )                                                     | 20         |
| disk.samples                                                                                           | 2          |
| cpu.sys.samples                                                                                        | 30         |
| cpu.usr                                                                                                | 98         |
| arc.backupmissing                                                                                      | 1          |
| cpu.io                                                                                                 | 20         |
| Archive Backup Delay Minutes ( archbackupdelaymins )                                                   | 40         |
| inventoryPurgeThreadInterval                                                                           | 720        |
| Age of Purging Collections (Hours) ( minFileAgeToPurge )                                               | 12         |
| cpu.idle.samples                                                                                       | 2          |
| unreachablenodeSleepTime                                                                               | 300        |
| cpu.idle                                                                                               | 20         |
| mem.swaptotal                                                                                          | 24         |
| TFA ISA CRS Profile Delay (minutes) ( tfaDbUtlCrsProfileDelay )                                        | 720        |
| cdb.backupmissing                                                                                      | 1          |
| cdb.backupmissing.samples                                                                              | 2          |
| Trim Size ( trimsize )                                                                                 | 500000     |
| Maximum Size of TFA Log (MB) ( maxLogSize )                                                            | 52428800   |
| minTimeForAutoDiagCollection                                                                           | 300        |
| skipScanThreshold                                                                                      | 100        |
| fileCountInventorySwitch                                                                               | 5000       |
| TFA ISA Purge Mode ( tfaDbUtlPurgeMode )                                                               | profile    |
| country                                                                                                | US         |
| Debug Mask (Hex) ( debugmask )                                                                         | 0x000000   |
| Setting for ACR redaction (none|SANITIZE|MASK) ( redact )                                              | none       |
| language                                                                                               | en         |
| AlertLogLevel                                                                                          | ALL        |
| BaseLogPath                                                                                            | ERROR      |
| encoding                                                                                               | UTF-8      |
| UserLogLevel                                                                                           | ALL        |
| Logs older than the time period will be auto purged(days[d]|hours[h]) ( manageLogsAutoPurgePolicyAge ) | 30d        |
| isaMode                                                                                                | enabled    |
'--------------------------------------------------------------------------------------------------------+------------'


cambio de ubicacion

Supongamos que queremos mover la ubicacion a un directorio compartido.
La ultima rama del repositorio debe de llamarse repository

tfactl set repositorydir=/mnt/tfa/repository 

Desinstalacion

Para su desinstalacion simplemente tenemos que ejecutarlo con el flag uninstall

[root@rac1 ~]$ which tfactl
/opt/app/12.1/grid/bin/tfactl
[root@rac1 ~]$  /opt/app/12.1/grid/bin/tfactl uninstall

Como siempre, mas infrmacion en oracle support: