Tag Archives: Kerberos

Easy setup of Kerberos Authentication for Oracle Databases

I have previously published a couple of blog posts related to Kerberos authentication for databases. In this post, I want to provide a simple, step-by-step tutorial for configuring Kerberos authentication. This tutorial is based on my lab setup within Oracle Cloud Infrastructure (OCI). Within this environment, I run both a database server and a corresponding Windows server configured as an Active Directory server. It should be noted that this tutorial is designed for a basic environment. The configuration must be adapted accordingly for special cases such as clusters, multiple AD forests or domains, OS configuration, etc.

Prerequisites and Requirements

Configuring Kerberos authentication for Oracle databases involves a number of tasks, each of which requires specific permissions. In my lab environment, of course, I have all the necessary permissions. In other environments, certain tasks may need to be delegated to other administrators. Essentially, the following steps need to be performed:

  • DB Server Install software component for Kerberos client tools as root user.
  • AD Server Create a service account in AD as a domain administrator.
  • DB Environment Configure the SQLNet environment as the Oracle user.
  • DB Instance Adjust the init.ora parameters and establish Kerberos accounts.

The subsequent Kerberos configuration relies on the following values:

  • AD Domain / KDC Realm: TRIVADISLABS.COM
  • AD Server / KDC: ad.trivadislabs.com (10.0.1.4)
  • Database Server (FQDN): db23.trivadislabs.com (10.0.1.23)
  • Database Server OS: Oracle Enterprise Linux 8 (Version: 8.8)
  • SamAccountName: db23
  • User Principal Name (UPN): db23.trivadislabs.com
  • Service Principle Name (SPN): oracle/db23.trivadislabs.com
  • Database SID: CDB23B with pluggable database PDB1B and PDB2B

Please note that for different environments and operating systems, the commands may need to be adjusted accordingly.

Step 1 Preparation Database Server

Command line commands to install Kerberos client utilities on OEL8/REL8

sudo dnf install krb5-workstation

Step 2 Service Account Configuration

The following steps should be performed on the AD server by a domain administrator or an administrative account with the required privileges. Essentially, the choice of the tool used for these tasks is not relevant; however, the following section describes only the relevant PowerShell commands.

If the service account already exists, we will delete it first. This step is optional. Nevertheless, it is not a bad idea to start the configuration from scratch. This will certainly make troubleshooting a bit easier.

$Hostname = "db23"
if (!(Get-ADUser -Filter "sAMAccountName -eq '$Hostname'")) {
  Write-Host "INFO : Service Account ($Hostname) does not exist."
} else  {
  Write-Host "INFO : Remove existing Service Account ($Hostname)."
  Remove-ADUser -Identity $Hostname -Confirm
} 

The PowerShell command provided below is used to create a service account with the appropriate flags set to support Kerberos encryption using both AES 128 and 256-bit methods. This guarantees that the keytab file can be generated with the necessary encryption types for AES, and authentication will function properly with such a keytab file. Update the service account name, password and UserDN accordingly.

$ServiceName = "db23"
$UserBaseDN  = "cn=Users," + (Get-ADDomain).DistinguishedName
$DNSRoot     = (Get-ADDomain).DNSRoot

# get the credentials for the kerberos service account
$credential = Get-Credential -message 'Kerberos Service Account' -UserName $ServiceName
$ServiceUserParams = @{
    Name                    =   $credential.UserName
    DisplayName             =   $ServiceName
    SamAccountName          =   $ServiceName
    UserPrincipalName       =   "oracle/$ServiceName.$DNSRoot"
    Description             =   "Kerberos Service User for $ServiceName"
    Path                    =   $UserBaseDN
    AccountPassword         =   $credential.Password
    PasswordNeverExpires    =   $true
    Enabled                 =   $true
    KerberosEncryptionType  =   "AES256"
}

# create kerberos service account
New-ADUser @ServiceUserParams

The final step on Windows involves creating a Service Principal Name (SPN) for the service user. If ktpass.exe is used to generate the keytab file, this is done automatically. However, as we are creating the keytab file on the database server using ktutil, we need to create the Service Principal Name (SPN) manually using setspn.

$ServiceName = "db23"
$DNSRoot     = (Get-ADDomain).DNSRoot
setspn $ServiceName -s oracle/$ServiceName.$DNSRoot

Example output of the command.

PS C:\Windows\system32> setspn $ServiceName -s oracle/$ServiceName.$DNSRoot@$Domain
Checking domain DC=trivadislabs,DC=com

Registering ServicePrincipalNames for CN=db23,CN=Users,DC=trivadislabs,DC=com
        oracle/db23.trivadislabs.com
Updated object
PS C:\Windows\system32>

Step 3 Oracle SQLNet Configuration

It is recommended to set up the SQLNet configuration for each database server in the $TNS_ADMIN directory. This is especially important if you are working with multiple Oracle Homes. Otherwise, multiple Kerberos configurations must be maintained for each database server.

Add the following kerberos configuration section to you sqlnet.ora file. Adjust the path to keytab and krb5.conf file accordingly.

# ----------------------------------------------------------------
# Kerberos settings
# ----------------------------------------------------------------
SQLNET.AUTHENTICATION_SERVICES=(beq,tcps,kerberos5pre,kerberos5)
SQLNET.AUTHENTICATION_KERBEROS5_SERVICE = oracle
SQLNET.FALLBACK_AUTHENTICATION = TRUE
SQLNET.KERBEROS5_KEYTAB = /u01/app/oracle/network/admin/krb5.keytab
SQLNET.KERBEROS5_CONF = /u01/app/oracle/network/admin/krb5.conf
SQLNET.KERBEROS5_CONF_MIT=TRUE

Create a new Kerberos configuration file, krb5.conf, in your $TNS_ADMIN folder. Adjust the KDC realm, domain, etc., as needed.

# ----------------------------------------------------------------
# OraDBA - Oracle Database Infrastructur and Security, 5630 Muri,
# Switzerland
# ----------------------------------------------------------------
# Name.......: krb5.conf
# Author.....: Stefan Oehrli (oes) stefan.oehrli@oradba.ch
# Editor.....: Stefan Oehrli
# Date.......: 2023.05.04
# Version....: --
# Purpose....: Kerberos Configuration File
# Notes......: --
# Reference..: Oracle Database Security Guide 19c
# ----------------------------------------------------------------
[libdefaults]
forwardable = true
default_realm = TRIVADISLABS.COM
 
[realms]
  TRIVADISLABS.COM = {
    kdc = ad.trivadislabs.com
  }
 
[domain_realm]
.trivadislabs.com = TRIVADISLABS.COM
trivadislabs.com = TRIVADISLABS.COM

It is advisable to restart both the listener and the databases afterward to ensure that the new sqlnet.ora configuration is applied universally. However, this restart does not necessarily need to occur immediately.

Step 4 Create keytab File

Generate a Ticket Granting Ticket (TGT) for the service principal. To confirm the service account and streamline the subsequent steps, obtain a TGT using okinit, an Oracle tool that relies on the previously mentioned sqlnet.ora configuration.

okinit db23@TRIVADISLABS.COM

Example output of the command.

oracle@db23:~/ [CDB23B] okinit db23@TRIVADISLABS.COM

Kerberos Utilities for Linux: Version 23.0.0.0.0 - Beta on 08-NOV-2023 16:17:13

Copyright (c) 1996, 2023 Oracle.  All rights reserved.

Configuration file : /u01/app/oracle/network/admin/krb5.conf.
Password for db23@TRIVADISLABS.COM:
oracle@db23:~/ [CDB23B] 

Obtain the Key Version Number (kvno) for the Service Principal. We need the key version number (kvno) for the service principal, which can be retrieved using the kvno utility. You also have the option to explicitly specify the ticket cache using the -c flag. The kvno is crucial for creating the keytab file.

By default, Linux Kerberos tools require a krb5.conf file in /etc. Since we do not intend to configure Kerberos for Linux authentication, we can specify the krb5.conf file from TNS_ADMIN by using the environment variable KRB5_CONFIG.

export KRB5_CONFIG=$TNS_ADMIN/krb5.conf
kvno -c /tmp/krb5cc_1000 db23@TRIVADISLABS.COM

Example output of the command.

oracle@db23:~/ [rdbms] export KRB5_CONFIG=$TNS_ADMIN/krb5.conf
oracle@db23:~/ [rdbms] kvno -c /tmp/krb5cc_1000 db23@TRIVADISLABS.COM
db23@TRIVADISLABS.COM: kvno = 2

We now create a keytab file with ktutil. The tool must be used interactively to read, create and write the keytab file. See the ktutil man page for full usage. In the following example, we use the aes256-cts-hmac-sha1-96 encryption type. Update the addent command accordingly with the correct kvno and encryption type. Optionally, you can add multiple encryption types to a keytab by running addent multiple times. The list of encryption types can be found at Kerberos Parameters. Make sure to use encryption types which are supported by your KDC.

oracle@db23:~/ [rdbms19] export KRB5_CONFIG=$TNS_ADMIN/krb5.conf
oracle@db23:~/ [rdbms19] mv $TNS_ADMIN/krb5.keytab $TNS_ADMIN/krb5.keytab.orig
oracle@db23:~/ [rdbms19] ktutil
ktutil:  addent -password -p oracle/db23.trivadislabs.com@TRIVADISLABS.COM -k 2 -e aes256-cts-hmac-sha1-96
Password for oracle/db23.trivadislabs.com@TRIVADISLABS.COM:
ktutil:  list -e
slot KVNO Principal
---- ---- ---------------------------------------------------------------------
   1    2 oracle/db23.trivadislabs.com@TRIVADISLABS.COM (aes256-cts-hmac-sha1-96)
ktutil:  wkt /u01/app/oracle/network/admin/krb5.keytab
ktutil:  q

Verify the new keytab file using oklist.

oracle@db23:~/ [rdbms] oklist -e -k

Kerberos Utilities for Linux: Version 23.0.0.0.0 - Beta on 08-NOV-2023 16:18:31

Copyright (c) 1996, 2023 Oracle.  All rights reserved.

Configuration file : /u01/app/oracle/network/admin/krb5.conf.
Keytab name: FILE:/u01/app/oracle/network/admin/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
   2 oracle/db23.trivadislabs.com@TRIVADISLABS.COM (AES-256 CTS mode with 96-bit SHA-1 HMAC) 

Step 5 Database Configuration

To enable Kerberos database authentication, it is necessary to modify the init.ora parameters os_authent_prefix=” and for system older als 23c also remote_os_authent=FALSE in each database. Both parameters require a database restart.

ALTER SYSTEM SET os_authent_prefix='' SCOPE=spfile;

Furthermore, it is essential to create or modify the appropriate database users for Kerberos authentication. In the following example, we will create a user named King and grant them the necessary privileges to establish a connection to the database and select information from V$SESSION.

ALTER SESSION SET CONTAINER=pdb1b;
CREATE USER king IDENTIFIED EXTERNALLY AS 'king@TRIVADISLABS.COM';
GRANT create session TO king;
GRANT SELECT ON v_$session TO king;

Step 6 Test Kerberos Authentication

We initially create a Ticket Granting Ticket (TGT) for a specific user, such as King.

okinit king

Example output of the command.

oracle@db23:~/ [CDB23B] okinit king

Kerberos Utilities for Linux: Version 23.0.0.0.0 - Beta on 03-NOV-2023 15:47:27

Copyright (c) 1996, 2023 Oracle.  All rights reserved.

Configuration file : /u01/app/oracle/network/admin/krb5.conf.
Password for king@TRIVADISLABS.COM: 

Now, we can connect directly to the PDB1B database using SQL*Plus without specifying a username and password.

oracle@db23:~/ [CDB23B] sqlplus /@pdb1b

SQL*Plus: Release 23.0.0.0.0 - Beta on Wed Nov 8 16:20:43 2023
Version 23.2.0.0.0

Copyright (c) 1982, 2023, Oracle.  All rights reserved.

Connected to:
Oracle Database 23c Enterprise Edition Release 23.0.0.0.0 - Beta
Version 23.2.0.0.0

SQL>

By querying the system context USERENV, you can find relevant information about the user, authentication method and more.

SET linesize 160 pagesize 200
COL db_user FOR A20
COL auth_method FOR A20
COL auth_id FOR A40

SELECT 
   sys_context('userenv','SESSION_USER') db_user,
   sys_context('userenv','AUTHENTICATION_METHOD') auth_method,
   sys_context('userenv','AUTHENTICATED_IDENTITY') auth_id
FROM dual;

Example output of the query.

SQL> SELECT
  2     sys_context('userenv','SESSION_USER') db_user,
  3     sys_context('userenv','AUTHENTICATION_METHOD') auth_method,
  4     sys_context('userenv','AUTHENTICATED_IDENTITY') auth_id
  5  FROM dual;

DB_USER 	     AUTH_METHOD	  AUTH_ID
-------------------- -------------------- ----------------------------------------
KING		     KERBEROS		  king@TRIVADISLABS.COM

Tips and Best Practices

Once Kerberos is up and running, the experience is generally smooth. However, it’s important to consider these best practices:

  • Start Simple: Begin with an uncomplicated setup. A multi-domain Oracle Maximum Availability Architecture (MAA) environment isn’t necessary for initial implementation.
  • Ensure Basic Infrastructure: Verify that the foundational configurations are in place, such as network access, open ports, proper name resolution, and synchronized time settings.
  • Avoid Ad-hoc Tweaks: If issues arise, resist the urge to make random changes. Certain details may be cached, complicating the troubleshooting process. Instead, systematically revert to the last working configuration and proceed cautiously.
  • Regenerate Keytab Files When Needed: Many issues can be resolved with a fresh and correctly configured keytab file. Don’t hesitate to recreate it.
  • Adopt Secure Practices Early: Begin with robust security measures, such as employing strong encryption algorithms and setting secure passwords. The mindset of ‘I’ll secure it later‘ often leads to vulnerabilities. 🤪
  • Document Troubleshooting Steps: Keeping a record of the steps and solutions can be invaluable for future reference.
  • Distinguish Between OS and DB Authentication: It’s critical to understand the differences between OS-level Kerberos authentication and Oracle database-specific Kerberos authentication. Do not confuse the two.

Common Errors and Troubleshooting

The Oracle Support Document 185897.1 Kerberos Troubleshooting Guide provides a comprehensive overview of potential issues you might encounter with database Kerberos authentication. Additionally, my blog post Kerberos Troubleshooting – A few approaches outlines practical troubleshooting examples. When addressing Kerberos authentication problems, enabling Oracle SQLNet Tracing is often indispensable; without it, you might find yourself groping in the dark.

Below a couple of common Database Kerberos Authentication issues

  • Incorrect Keytab File: The keytab file may be outdated or incorrectly configured.
  • Service Principal Name (SPN) Mismatches: The SPN registered in Active Directory doesn’t match the one the Oracle server is expecting.
  • Clock Skew: There’s too much time difference between the client and server machines, or between the server and the domain controller.
  • DNS Resolution Problems: The client or server may be unable to resolve the domain names to their IP addresses.
  • Expired Credentials: User credentials or service tickets may have expired.
  • Kerberos Realm Confusion: Incorrect configuration of the Kerberos realm can lead to failed authentication.
  • Version Mismatch: The version of Kerberos on the client does not match with what the Oracle Database expects.
  • Access Denied: Improper permissions set for the Oracle service account within Active Directory.
  • SQLNet Configuration: sqlnet.ora or krb5.conf file may have incorrect entries or lack necessary Kerberos parameters.
  • Kerberos Ticket Issues: Problems obtaining or using a valid Kerberos ticket due to cache issues or misconfigurations.
  • Network Issues: Latency or connectivity problems can prevent proper communication between the client, server, and Kerberos Key Distribution Center (KDC).
  • Case Sensitivity: Kerberos is case-sensitive; mismatches in case between configurations can cause failures.
  • Client Configuration Errors: The Kerberos client may not be configured correctly on the user’s machine, leading to authentication errors.
  • Multi-Domain Environments: Additional complexities when the database and users are in different domains or forests.
  • KVNO Mismatch: Discrepancies between the KVNO in the keytab file and the KVNO for the service principal in the KDC can result in authentication failures. This often happens after a password change for a service account where the keytab file was not simultaneously updated

Unfortunately, I keep running into a new problem every time I try to configure Kerberos. When writing this blog post, it took me a relatively long time to figure out that the User Principal Name (UPN) of my service account was not set. The error was of course an ORA-01017 and ORA-12631, although this can easily be checked with the following LDAP query.

ldapsearch -h ad.trivadislabs.com -p 389 \
-D king@TRIVADISLABS.COM -q \
-b "cn=Users,dc=trivadislabs,dc=com" \
-s sub "(sAMAccountName=db23)" \
userPrincipalName servicePrincipalName

Conclusion

As you can see, it is clear that setting up and configuring Kerberos is a straightforward process. It provides a relatively simple way to increase the security of database accounts and at the same time significantly improve the user-friendliness of single sign-on (SSO). However, the devil is in the detail. In complex Active Directory domains or Key Distribution Centers (KDCs), additional configuration, such as setting up domain trust, can involve a certain amount of complexity. Furthermore, not all tools and clients are Kerberos-capable out of the box. Therefore, it is important to understand the database users and their access methods. Even with the integration of Kerberos, a well thought-out user and role concept remains essential. However, Kerberos integrates seamlessly with Oracle’s Centrally Managed Users (CMU) and can coexist with other authentication methods, such as password-based authentication. Why don’t you start by configuring Kerberos for your DBAs and power users?

Additional Resources

Some links and references related to this topic.

Simplified Keytab creation using Linux Tools #JoelKallmanDay

Today’s  #JoelKallmanDay, my topic is about simplifying the configuration of Kerberos authentication of Oracle databases using Linux tools. I have already written a few things about Kerberos in the past. The blog posts on this topic are usually tagged with Kerberos. Today I want to show you an alternative method how to create the keytab file directly on the database server without the help of a domain admin. This is especially helpful in larger environments or when troubleshooting. You have a new keytab file immediately, without having to first create an incident ticket and wait for support from your Windows administration colleagues. As a rule, they are not bored either and are glad to have one task less.

Some Basics

Kerberos is a networked authentication system that Oracle uses authenticate Oracle Database users. The following graphic shows the Kerberos authentication process at a glance. The complete configuration is described in detail in Oracle® Database Security Guide 23c – Configuring Kerberos Authentication or Oracle Support Document 1996329.1. Further links can be found in the Reference chapter. This blog post is only about an alternative creation of the keytab file.

The Kerberos authentication Process at a Glance

The keytab file plays a vital role in securely managing service keys, especially service principals, for the host’s various services, including those associated with Kerberos authentication for the Oracle database. This important file equips the Oracle database with the necessary information to interact with the Key Distribution Center (KDC) and perform user authentication.

Traditionally, the keytab file is generated on the KDC server. In the context of Windows Active Directory, the ktpass tool stands as the go-to utility for this task. Detailed guidance and examples for using ktpass can be found in official documentation.

For instance, here’s an example of how I create a keytab file using ktpass in my test lab. In this case, I’m associating the service principal with the user DB19. The resulting keytab file includes all available cryptographic keys for enhanced security.

ktpass.exe -princ oracle/db19.trivadislabs.com@TRIVADISLABS.COM
  -mapuser db19 -pass <PASSWORD> -crypto AL
  -ptype KRB5_NT_PRINCIPAL
  -out C:\stage\db19.trivadislabs.com.keytab

In a test lab you can do everything yourself. This is not possible in productive environments. You need admin access to the Active Directory or have a Windows admin at hand who can do it. Afterwards you have to copy the file partially over detours on the database server. The whole thing is cumbersome and error-prone.

Requirements

The following requirements must be met in order to use the Linux tools for configuration:

  • Tools Installing Kerberos client utilities on the database server. Whereas we don’t do Kerberos configuration for the operating system, we just use the tools. Kerberos is only used for database authentication in this example.
  • AD User Creating a service account in Active Directory and set the service principal name (SPN) for this account.
  • Configuration have the basic Kerberos configuration ready in your database environment e.g., $TNS_ADMIN/sqlnet.ora and $TNS_ADMIN/krb5.conf

Example of Tools Installation

Command line commands to install Kerberos client utilities on OEL8

sudo dnf install krb5-workstation

Command line commands to install Kerberos client utilities on OEL7

sudo yum -y install krb5-workstation

Example of AD User Creation

PowerShell commands to create service principal account with the flags set for This account supports Kerberos AES 128 bit encryption and This account supports Kerberos AES 256 bit encryption. This ensures that the keytab file can be created with the encryption type for AES respectively that an authentication then also works with such a keytab file. Additionaly we do set the SPN using setspn.

$Hostname = "db19"
$sPWD = ConvertTo-SecureString -AsPlainText "<PASSWORD>" -Force
$UsersDN  = "cn=Users," + (Get-ADDomain).DistinguishedName
$DNSRoot  = (Get-ADDomain).DNSRoot
$Domain   = (Get-ADDomainController).Domain.ToUpper()

if (!(Get-ADUser -Filter "sAMAccountName -eq '$Hostname'")) {
  Write-Host "INFO : User does not exist."
} else  {
  Write-Host "INFO : Remove existing User."
  Remove-ADUser -Identity $Hostname -Confirm
} 

Write-Host "INFO : Create service account for DB server $Hostname."
New-ADUser -SamAccountName $Hostname -Name $Hostname
  -DisplayName $Hostname
  -UserPrincipalName "oracle/$Hostname.$DNSRoot"
  -Description "Kerberos Service User for $Hostname"
  -Path $UsersDN -AccountPassword $sPWD
  -Enabled $true
  -KerberosEncryptionType "AES128, AES256"

Additionaly we do set the SPN using setspn.

setspn $Hostname -s oracle/$Hostname.$DNSRoot@$Domain
PS C:\Windows\system32> setspn $Hostname -s oracle/$Hostname.$DNSRoot@$Domain
Checking domain DC=trivadislabs,DC=com

Registering ServicePrincipalNames for CN=db19,CN=Users,DC=trivadislabs,DC=com
        oracle/db19.trivadislabs.com@TRIVADISLABS.COM
Updated object

Example Kerberos Configuration

Basic krb5.conf file in the $TNS_ADMIN folder. This example does configure ad.trivadislabs.com as KDC for the realm / domain TRIVADISLABS.COM.

# ----------------------------------------------------------------
# OraDBA - Oracle Database Infrastructur and Security, 5630 Muri,
# Switzerland
# ----------------------------------------------------------------
# Name.......: krb5.conf
# Author.....: Stefan Oehrli (oes) stefan.oehrli@oradba.ch
# Editor.....: Stefan Oehrli
# Date.......: 2023.05.04
# Version....: --
# Purpose....: Kerberos Configuration File
# Notes......: --
# Reference..: Oracle Database Security Guide 19c
# ----------------------------------------------------------------
[libdefaults]
forwardable = true
default_realm = TRIVADISLABS.COM
 
[realms]
  TRIVADISLABS.COM = {
    kdc = ad.trivadislabs.com
  }
 
[domain_realm]
.trivadislabs.com = TRIVADISLABS.COM
trivadislabs.com = TRIVADISLABS.COM

Extract from sqlnet.ora in $TNS_ADMIN with the Kerberos configuration

# ----------------------------------------------------------------
# Kerberos settings
# ----------------------------------------------------------------
SQLNET.AUTHENTICATION_SERVICES=(beq,tcps,kerberos5pre,kerberos5)
SQLNET.AUTHENTICATION_KERBEROS5_SERVICE = oracle
SQLNET.FALLBACK_AUTHENTICATION = TRUE
SQLNET.KERBEROS5_KEYTAB = /u01/app/oracle/network/admin/krb5.keytab
SQLNET.KERBEROS5_CONF = /u01/app/oracle/network/admin/krb5.conf
SQLNET.KERBEROS5_CONF_MIT=TRUE

Using the Kerberos Utilities

Lets use the different commandline utilities to create the keytab file on the database server as user oracle.

Step 1: Create a TGT for the service principal

To verify the service account and simplify the following steps we do get a ticket granting ticket (TGT) using okinit. Whereby okinit is an Oracle tool and does require the sqlnet.ora configuration mentioned before.

oracle@db19:~/ [rdbms19] okinit db19

Kerberos Utilities for Linux: Version 19.0.0.0.0 - Production on 11-OCT-2023 21:33:35

Copyright (c) 1996, 2019 Oracle.  All rights reserved.

Configuration file : /u01/app/oracle/network/admin/krb5.conf.
Password for db19@TRIVADISLABS.COM:

Step 2: Check the ticket cache

Verify the ticket cache using oklist

oracle@db19:~/ [rdbms19] oklist

Kerberos Utilities for Linux: Version 19.0.0.0.0 - Production on 11-OCT-2023 21:34:54

Copyright (c) 1996, 2019 Oracle.  All rights reserved.

Configuration file : /u01/app/oracle/network/admin/krb5.conf.
Ticket cache: FILE:/tmp/krb5cc_1000
Default principal: db19@TRIVADISLABS.COM

Valid starting     Expires            Service principal
10/11/23 21:33:39  10/12/23 07:33:39  krbtgt/TRIVADISLABS.COM@TRIVADISLABS.COM
	renew until 10/12/23 21:33:35

Step 3: Get the kvno for the Service Principle

We need the key version number (kvno) of the service principle. this can be queried using the kvno utility. Verify above which ticket cache is used. Optionally specify the ticket cache explicitly using -c. The kvno will be used when creating the keytab file.

oracle@db19:~/ [rdbms19] kvno -c /tmp/krb5cc_1000 db19@TRIVADISLABS.COM 
db19@TRIVADISLABS.COM: kvno = 2

Step 4: Create a keytab file using ktutil

We now create a keytab file with ktutil. The tool must be used interactively to read, create and write the keytab file. See the ktutil man page for full usage. In the following example, we use the aes256-cts-hmac-sha1-96 encryption type. Update the addent command accordingly with the correct kvno and encryption type. Optionally, you can add multiple encryption types to a keytab by running addent multiple times. The list of encryption types can be found at Kerberos Parameters. Make sure to use encryption types which are supported by your KDC.

oracle@db19:~/ [rdbms19] mv $TNS_ADMIN/krb5.keytab $TNS_ADMIN/krb5.keytab.orig
oracle@db19:~/ [rdbms19] ktutil
ktutil:  addent -password -p oracle/db19.trivadislabs.com@TRIVADISLABS.COM -k 2 -e aes256-cts-hmac-sha1-96
Password for oracle/db19.trivadislabs.com@TRIVADISLABS.COM:
ktutil:  list -e
slot KVNO Principal
---- ---- ---------------------------------------------------------------------
   1    2 oracle/db19.trivadislabs.com@TRIVADISLABS.COM (aes256-cts-hmac-sha1-96)
ktutil:  wkt /u01/app/oracle/network/admin/krb5.keytab
ktutil:  q

Step 5: Verify the new keytab File

Verify the new keytab file using oklist

oracle@db19:~/ [rdbms19] oklist -e -k

Kerberos Utilities for Linux: Version 19.0.0.0.0 - Production on 11-OCT-2023 22:41:00

Copyright (c) 1996, 2019 Oracle.  All rights reserved.

Configuration file : /u01/app/oracle/network/admin/krb5.conf.
Keytab name: FILE:/u01/app/oracle/network/admin/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
   2 oracle/db19.trivadislabs.com@TRIVADISLABS.COM (AES-256 CTS mode with 96-bit SHA-1 HMAC)

Conculsion

Especially in large environments, where you sometimes have to wait several days for a service ticket to be processed, the alternative method for creating a keytab file is a relief. The keytab file is immediately where you need it. No need for cumbersome copying via SSH, fileshare, tunnels etc. You need another or an additional encryption type in the keytab file? Nothing simpler than that. One call of ktutil and addent and you have an additional entcryption type in the keytab file.

Have fun configuring Kerberos

References

Some links related to this topic.

  • Oracle® Database Security Guide 23c – Configuring Kerberos Authentication
  • Oracle Support Document 1375853.1 Primary Note For Kerberos Authentication
  • Oracle Support Document 1996329.1 How To Configure Kerberos Authentication In A 12c Database
  • Oracle Support Document 1304004.1 Configuring Kerberos Authentication with a Microsoft Windows Active Directory KDC
  • Oracle Support Document 132804.1 Enabling Kerberos Authentication
  • Oracle Support Document 185897.1 Kerberos Troubleshooting Guide
  • Oracle Support Document 1523651.1 Kerberos Authentication With Oracle JDBC Thin Driver And Microsoft Active Directory
  • Oracle Support Document 1609359.1 How To Use Kerberos Authentication to connect to a database with SQL Developer with thin JDBC
  • Oracle Support Document 294136.1 Kerberos: High Level Introduction and Flow
  • Microsoft Windows Server Documentation ktpass
  • Linux Man Pages ktutil
  • Linux Man Pages kvno
  • IANA Kerberos Encryption Type Numbers
  • OraDBA Kerberos related blog posts see Kerberos

Kerberos Troubleshooting – A few approaches

It is way too long ago since my last blog post. These were or are busy weeks for me. Any way, I finally found some time to start writing a blog post about a special setup for kerberos authentication of Oracle databases. It is about configuring kerberos authentication for multiple database servers with only one active directory account and corresponding Service Priciple Names (SPN). Additionally there is an challenge, that the keytab file should only be created with ktutil directly on the DB server. Access to a Windows server and use of ktpass.exe is not possible. I did setup a nice test case on a couple of compute instances on Oracle cloud infrastructure. During the verification of the test setup I had to realise that the kerberos authentication does not work as planned. Until now it is not possible to create a keytab file with ktutil that I can use successfully with Active Directory. The same kerberos configuration with a keytab create with ktpass.exe on the AD server does work. But that’s on other story…

The aim of this blog post is to sum up a couple of troubleshooting actions I came across. Kerberos itself is around since a couple of decades. Therefore you will find various documentation, RFC, etc. But it is not always easy to recognise what is still relevant and what not. Mainly because the implementation of Kerberos at both Oracle and Microsoft is not necessarily the same or 100% MIT Kerberos compliant. The fact that there are different versions of Oracle, MS AD and Kerberos makes it even more exciting 🙂

Basics

A basic requirement for Kerberos is the network and time configuration.

  • Problem: okinit does fail with clock skew too great
  • Cause: The systems involved must be synchronous in terms of system time e.g. using a NTP service to configure date / time. If the system times differ to much you will receive this error when using okinit.
  • Solution: Configure proper system times using NTP service. Small time drifts can be covered by setting SQLNET.KERBEROS5_CLOCKSKEW=300 in sqlnet.ora
  • Problem: Miscellaneous errors due to wrong / missing network configuration.
  • Cause: Using CNAME rather A records, no DNS configuration, no revers lookkup etc
  • Solution: Configure proper DNS name resolution for database service as well MS active directory service. Each system must be able to be resolved by name or IP address. Kerberos will look for service principle names based on A records.
oracle@db:/u00/app/oracle/network/admin/ [TDB190S] cd
oracle@db:~/ [TDB190S] nslookup win2016ad.trivadislabs.com
Server:		10.0.1.4
Address:	10.0.1.4#53

Name:	win2016ad.trivadislabs.com
Address: 10.0.1.4

oracle@db:~/ [TDB190S] nslookup 10.0.1.4
4.1.0.10.in-addr.arpa	name = win2016ad.trivadislabs.com.

oracle@db:~/ [TDB190S] nslookup db
Server:		10.0.1.4
Address:	10.0.1.4#53

db.trivadislabs.com	canonical name = ol7db19.trivadislabs.com.
Name:	ol7db19.trivadislabs.com
Address: 10.0.1.6

oracle@db:~/ [TDB190S] nslookup 10.0.1.6
6.1.0.10.in-addr.arpa	name = ol7db19.trivadislabs.com.

Trace and Log Files

Kerberos Trace

As of Oracle 12c release 2 it is possible to enable kerberos tracing by setting KRB5_TRACE to a trace file. This logs the Kerberos calls in the current session.

export KRB5_TRACE=/u00/app/oracle/network/admin/kerberos.trc
oracle@db:~/ [TDB190S] okinit king

Kerberos Utilities for Linux: Version 19.0.0.0.0 - Production on 08-JUN-2020 20:54:16

Copyright (c) 1996, 2019 Oracle.  All rights reserved.

Configuration file : /u00/app/oracle/network/admin/krb5.conf.
Password for king@TRIVADISLABS.COM:

A sample output of a kerberos trace file:

oracle@db:~/ [TDB190S] head -10 /u00/app/oracle/network/admin/kerberos.trc
[5645] 1591649656.590082: Getting initial credentials for king@TRIVADISLABS.COM
[5645] 1591649656.590084: Sending unauthenticated request
[5645] 1591649656.590085: Sending request (199 bytes) to TRIVADISLABS.COM
[5645] 1591649656.590086: Resolving hostname ad.trivadislabs.com
[5645] 1591649656.590087: Sending initial UDP request to dgram 10.0.1.4:88
[5645] 1591649656.590088: Received answer (196 bytes) from dgram 10.0.1.4:88
[5645] 1591649656.590089: Sending DNS URI query for _kerberos.TRIVADISLABS.COM.
[5645] 1591649656.590090: No URI records found
[5645] 1591649656.590091: Sending DNS SRV query for _kerberos-master._udp.TRIVADISLABS.COM.
[5645] 1591649656.590092: Sending DNS SRV query for _kerberos-master._tcp.TRIVADISLABS.COM.

Oracle SQLNet tracing

For Kerberos troubleshooting with Oracle SQLNet it is helpful to disable ADR tracing. Not mandatory, but makes life a bit easier. Set DIAG_ADR_ENABLED in sqlnet.ora to OFF.

DIAG_ADR_ENABLED=OFF

Before KRB5_TRACE was available, okinit calls could only be traced with sqlnet.ora and TRACE_LEVEL_OKINIT. See also MOS note 162668.1. The parameter does not make sense when you already use KRB5_TRACE.

TRACE_LEVEL_OKINIT=SUPPORT
TRACE_DIRECTORY_OKINIT=/u00/app/oracle/network/
TRACE_FILE_OKINIT=okinit.trc

For further analysis you usually have to switch on SQLNet Tracing. Don’t even thing about setting an other level than SUPPORT (16). Kerberos calls are only available with the highest level.

TRACE_LEVEL_OKINIT=SUPPORT
TRACE_DIRECTORY_OKINIT=/u00/app/oracle/network/
TRACE_FILE_OKINIT=okinit.trc

Enable tracing for SQLNet clients:

TRACE_LEVEL_CLIENT=SUPPORT
TRACE_DIRECTORY_CLIENT= /u00/app/oracle/network/trc
TRACE_FILE_CLIENT=sqlnet_client.trc

Enable tracing for SQLNet Server:

TRACE_LEVEL_SERVER=SUPPORT
TRACE_DIRECTORY_SERVER= /u00/app/oracle/network/trc
TRACE_FILE_SERVER=sqlnet_server.trc

The errors in the trace files are not always obvious. You can find a few infos and hint in MOS note 185897.1. But most of the time there is no way around searching for the corresponding error or function call in Oracle Support or the search engine of choice.

Network Tracing

The next level is to trace the network calls. Depending on the environment you can directly use Wireshark. But it is much easier to first create a network dump via command line and to analyse it later using Wireshark. I use tcpdump on my OCI environment and download the trace file to my MacBook, where I then use Wireshark.

Get the available interfaces:

sudo tcpdump -D
1.nflog (Linux netfilter log (NFLOG) interface)
2.nfqueue (Linux netfilter queue (NFQUEUE) interface)
3.usbmon1 (USB bus number 1)
4.ens3
5.any (Pseudo-device that captures on all interfaces)
6.lo [Loopback]

Start tracing for interface ens3:

sudo tcpdump -i ens3 -s 65535 -w /tmp/network_okcreate.trc

Keep it running until while testing the kerberos authentication. As soon as done copy the trace file to the client an open it using Wireshark. The following picture does show a trace dump where the kerberos protocol has been selected.

Wireshark sample output

A part of the kerberos packet is encrypted and not visible as you can see in following picture.

Wireshark enc

Kerberos does use the service’s secret key to encrypt these messages. You can import the keytab file into Wireshark to decrypt the messages. For this purpose the keytab file must be specified in Wireshark in the preferences. Click Edit > Preferences > Protocols > KRB5.

Wireshark Preferences

You now see the message content of the packet. This is in particular useful when you have to analyse issues related to ticket size, missing groups etc.

Wireshark decrypted

Conclusion

Unfortunately my Kerberos problem is still not solved. Nevertheless I did get the opportunity to practice a couple of Kerberos tracing methods. The introduction of KRB5_TRACE did simplify tracing a bit, but in most case you still have to use SQLNet or network tracing to find the root cause of you Kerberos problem. A direct solution is unfortunately not always found with tracing. At least you have all the relevant information to search My Oracle Support, open a service request or try your luck at googling for a solution.

Good luck with your Kerberos setup. 😉

References

Some links related to this blog post:

  • Kerberos Troubleshooting Guide [185897.1]
  • Master Note For Kerberos Authentication [1375853.1]
  • How to Trace Unix System Calls [110888.1]
  • Tracing Okinit [162668.1]
  • How to Enable Oracle SQL*Net Client, Server, Listener, Kerberos and External procedure Tracing from Net Manager [395525.1]
  • Requesting kerberos TGT with OKINT errors with okinit: Clock skew too great in 12.1.0.2 [2312008.1]

Using Kerberos in Oracle Standard Edition

Since the release of Oracle 12cR1 mid 2013 the network encryption and strong authentication services has been removed from the Oracle Advanced Security Option. Both feature are now available for any licensed editions. Corresponding section in the Oracle Licensing Guide for 11g R2 and 12c R1 has been updated.

Network encryption (native network encryption and SSL/TLS) and strong authentication services (Kerberos, PKI, and RADIUS) are no longer part of Oracle Advanced Security and are available in all licensed editions of all supported releases of the Oracle database.

Oracle Network Encryption does work quite well for Standard or Enterprise Edition even with the Instant Client. But there is no Kerberos support available Oracle 11.2.0.4 Standard Edition or any other 11g SE release. This can be verified on the command line with the command adapters:

oracle@urania:/u00/app/oracle/product/11.2.0.4se/lib/ [TDB11B] adapters

Installed Oracle Net transport protocols are:

    IPC
    BEQ
    TCP/IP
    SSL
    RAW
    SDP/IB

Installed Oracle Net naming methods are:

    Local Naming (tnsnames.ora)
    Oracle Directory Naming
    Oracle Host Naming
    Oracle Names Server Naming

Installed Oracle Advanced Security options are:

    RC4 40-bit encryption
    RC4 56-bit encryption
    RC4 128-bit encryption
    RC4 256-bit encryption
    DES40 40-bit encryption
    DES 56-bit encryption
    3DES 112-bit encryption
    3DES 168-bit encryption
    AES 128-bit encryption
    AES 192-bit encryption
    AES 256-bit encryption
    MD5 crypto-checksumming
    SHA-1 crypto-checksumming

There is a MOS Note 2145731.1 which describes how to enable the Radius adapter. The same method can be used to enable Kerberos as well, event if an other MOS Note 2028070.1 specifies that Kerberos is not available in Standard Edition.

Radius and Kerberos adapters is part of the object file nautab.o. In $ORACLE_HOME/lib are two object files. nautab.o and the nautab_ee.o.dbl:

oracle@urania:~/ [TDB11B] ls -al $ORACLE_HOME/lib/naut*
-rw-r--r--. 1 oracle users 4864 Jul 15  2013 /u00/app/oracle/product/11.2.0.4se/lib/nautab_ee.o.dbl
-rw-r--r--. 1 oracle users 4520 Jul  1 18:01 /u00/app/oracle/product/11.2.0.4se/lib/nautab.o

To get Kerberos support you just have to replace the nautab.o with nautab_ee.o.dbl…:

oracle@urania:~/ [TDB11B] cp $ORACLE_HOME/lib/nautab.o $ORACLE_HOME/lib/nautab_se.o.dbl
oracle@urania:~/ [TDB11B] cp $ORACLE_HOME/lib/nautab_ee.o.dbl $ORACLE_HOME/lib/nautab.o

oracle@urania:~/ [TDB11B] ls -al $ORACLE_HOME/lib/naut*
-rw-r--r--. 1 oracle users 4864 Jul 15  2013 /u00/app/oracle/product/11.2.0.4se/lib/nautab_ee.o.dbl
-rw-r--r--. 1 oracle users 4864 Jul  1 19:54 /u00/app/oracle/product/11.2.0.4se/lib/nautab.o
-rw-r--r--. 1 oracle users 4520 Jul  1 19:54 /u00/app/oracle/product/11.2.0.4se/lib/nautab_se.o.dbl

… and relink the binaries:

oracle@urania:~/ [TDB11B] relink all
writing relink log to: /u00/app/oracle/product/11.2.0.4se/install/relink.log

As you can see the command adapters now shows Radius and Kerberos adapters:

oracle@urania:~/ [TDB11B] adapters

Installed Oracle Net transport protocols are:

    IPC
    BEQ
    TCP/IP
    SSL
    RAW
    SDP/IB

Installed Oracle Net naming methods are:

    Local Naming (tnsnames.ora)
    Oracle Directory Naming
    Oracle Host Naming
    Oracle Names Server Naming

Installed Oracle Advanced Security options are:

    RC4 40-bit encryption
    RC4 56-bit encryption
    RC4 128-bit encryption
    RC4 256-bit encryption
    DES40 40-bit encryption
    DES 56-bit encryption
    3DES 112-bit encryption
    3DES 168-bit encryption
    AES 128-bit encryption
    AES 192-bit encryption
    AES 256-bit encryption
    MD5 crypto-checksumming
    SHA-1 crypto-checksumming
    Kerberos v5 authentication
    RADIUS authentication

As of now kerberos can be configured and used as usual. If for a reason okinit and oklist are also required, they must be copied from an existing Oracle 11g Enterprise Edition installation. By default they are not part of a Standard Edition. So let’s copy the binaries and the corresponding message files:

] cp 11.2.0.4/bin/okinit 11.2.0.4se/bin/okinit 
oracle@urania:/u00/app/oracle/product/ [TDB11B] cp 11.2.0.4/bin/oklist 11.2.0.4se/bin/oklist
oracle@urania:/u00/app/oracle/product/ [TDB11B] cp 11.2.0.4/bin/okinit 11.2.0.4se/bin/okinit
oracle@urania:/u00/app/oracle/product/ [TDB11B] cp 11.2.0.4/bin/okdstry 11.2.0.4se/bin/okdstry
oracle@urania:/u00/app/oracle/product/ [TDB11B] cp 11.2.0.4/network/mesg/naukus.msb 11.2.0.4se/network/mesg/naukus.msb
oracle@urania:/u00/app/oracle/product/ [TDB11B] cp 11.2.0.4/network/mesg/naukus.msg 11.2.0.4se/network/mesg/naukus.msg

Now you can use Kerberos as you want. You just have to struggle with the usual kerberos issues and bugs 🙂 But more on them here kerberos.

References

Some links related to this topic.

  • Oracle® Database Licensing Information User Manual 12c Release 1 (12.1) Oracle Advanced Security
  • Oracle® Database Licensing Information 11g Release 2 (11.2) Oracle Advanced Security
  • How To Enable Radius Adapter In Oracle Database 11g Standard Edition [2145731.1]
  • Kerberos Authentication On Standard Edition [2028070.1]
  • Configuring ASO Kerberos Authentication with a Microsoft Windows 2008 R2 Active Directory KDC [1304004.1]