PKI Authentication in CloudForms 4.1 (via integration with Red Hat IdM)

More below

Hello reader, and welcome to my first post for the CloudForms Now blog! Today I'll be walking you through the configuration of certificate based authentication and authorization in CloudForms 4.1. In addition to the steps for configuring CloudForms for PKI, I will also include instructions for creating user certificates in Red Hat Identity Management.

First, before we jump into the technical details, let's go over a few use cases for this setup and review some of the benefits.

Some of the standard use cases are:

  • Environments with an existing PKI infrastructure and a requirement for applications to use it for authentication
  • Military Common Access Card (CAC) authentication
  • Enterprise smart card authentication

The use case that I will be addressing is the first one, but only slight modifications are required to address the other two. The instructions that I am providing assume that the enterprise PKI provider is Red Hat Identity Management.

The advantages of this configuration are the same standard benefits that you would expect from any certificate based authentication setup (better security, simplified user management, simplified two factor authentication), but additionally, it also provides improved auditing, since every request to the web server is attributable to a specific user.

Here is a quick overview of the authentication workflow:

  1. The user opens the CloudForms URL in their browser
  2. CloudForms automatically redirects the user to a predefined login page
  3. An SSL handshake takes place and the user's cert is checked against the Red Hat Identity Management CA (or an enterprise CA), and an OCSP check is performed to verify that the user's certificate is not revoked
  4. After authentication, the user's information is passed to CloudForms
  5. CloudForms checks the user's group membership, and if the user belongs to a group with an associated role in CloudForms, they are logged in with that role. (Note: If the user has never logged into CloudForms before, but they belong to a group with an associated role, their user is created and the role is assigned.)

So, now that you know the what and the why, let's get into the how!

To begin, you will need the following:

  • A working Red Hat IdM or FreeIPA server
  • A working CloudForms 4.1 appliance
  • Forward and reverse DNS entries for each (or /etc/hosts entries in each server)

First, let's go over the steps for creating a user certificate in Red Hat Identity Management (referred to as IdM from this point forward). In this example, we are assuming that the user we are creating a certificate for does not exist yet in IdM. Screenshots for some of these steps are included for reference.

  1. Using openssl, create a CSR for the client certificate (as well as a key). Remember the location of this key, as you will need it later to convert your x509 certificate to p12. The easiest way to keep track of this is to run these commands from the root home directory of the IdM server. The user in this example will be john.
    [root@idm ~]# openssl req -out john.csr -new -newkey rsa:2048 -nodes -keyout john.key
  2. Use the cat command to output the CSR, and copy the output. Don't forget to include the lines -----BEGIN CERTIFICATE----- and -----END CERTIFICATE-----.
  3. Create a new user from the IdM web interface.
    • For this example, we will create the user john who we created the certificate request for in the previous steps.
    • Under "Identity -> Users -> Active Users" click the "+Add" button located above the table of users. (This is the landing page by default in IdM, so you will be on the correct page immediately after logging in.)
    • Fill in the "User login" (john), "First name" (John), and "Last name" (Shrader) fields.
    • Click "Add and Edit" at the bottom of the screen.
  4. Create an x509 certificate for the user john with the CSR copied in step 2. idm-user-cert-img-1
    • In the "User: john" screen, click the "Actions" drop down near the top of the page and choose "New certificate".
    • In the "New certificate" window, choose "IECUserRoles" from the drop down.
    • Paste the CSR you copied earlier into the window.
    • Click "Issue"
  5. Prepare the x509 certificate for conversion (via the CLI).
    • After the certificate has been successfully issued, scroll down to "Certificate" and click "Show". idm-user-cert-img-2
    • Copy the certificate that is displayed. idm-user-cert-img-3
    • On the IdM server, run the following command: echo "<pasted certificate>" | fold -w 64 > john.crt (full example can be seen in footnote). 1
    • Note: If you do not pipe the certificate to fold -w 64, it will not be formatted correctly and all of the following steps will fail.
    • Use VIM to edit the certificate and add the following
      • At the top, add a new line and fill it in with five dashes, the words BEGIN CERTIFICATE in all caps, then five more dashes (with no spaces between the dashes and the words). -----BEGIN CERTIFICATE-----
      • At the bottom, add another line below the certificate information and add five dashes, the words END CERTIFICATE in all caps, and five more dashes. -----END CERTIFICATE-----
      • The end result should be a properly formatted certificate.
  6. Convert the x509 certificate into a .p12 certificate for PKI client authentication.
    • Using openssl, convert the x509 certificate to a .p12 (you must have the original key available as well). [root@idm ~]# openssl pkcs12 -export -out john.p12 -inkey john.key -in john.crt -certfile john.crt
      • You can choose an export password for the certificate, or leave it blank.
    • Using your SCP tool of choice, move the .p12 certificate to the client machine where you plan to use it. idm-user-cert-img-4
    • Import the .p12 certificate in your browser and begin testing. idm-user-cert-img-5

Now that we have a client certificate for testing, let's configure CloudForms to use it for authentication!

Before we begin, it's a good idea to tail these 4 files prior to starting, and keep them running the entire time. This will make it easier to troubleshoot any issues you may encounter.

# tail -f /var/www/miq/vmdb/log/evm.log 
# tail -f /var/www/miq/vmdb/log/apache/ssl_error.log 
# tail -f /var/www/miq/vmdb/log/apache/ssl_access.log 
# tail -f /var/www/miq/vmdb/log/apache/ssl_request.log

Now that we have those running, let's begin our configuration.

  1. Configure your appliance (in the UI) for httpd external authentication.
    • Choose from the sidebar "Settings -> Configure".
    • Under "Settings", expand CFME Region, Zones, Zone: Default zone (current).
    • Choose "Server: EVM ######".
    • In the tabs at the top of the EVM screen, choose "Authentication".
    • Configure the screen as shown in the screenshot. appliance-ui-img-1
      • In the "Mode" drop down, choose "External (httpd)".
      • Click the check boxes for "Enable single sign on", "Disable local login", and "Get User Groups from External Authentication (httpd)".
    • Note: DO NOT click "Enable SAML" while the "Disable Local Login" box is checked. Doing so will redirect authentication requests and you will have to manually repair this from the command line. The "Enable SAML" box should never be checked when using PKI auth.
  2. Configure your appliance for external authentication through the appliance_console.
    • Begin by typing the appliance_console command, then follow the steps documented in the screenshots.
    • Please note that IdM is the only option here; this step is configuring the ipa-client in the background.
    • appliance-console-img-1
    • Note: As your appliance is updated, please be aware that the appliance_console option numbers may change. Choose "Configure External Authentication (httpd)" regardless of the option number.
  3. Update group/role relationships in CloudForms to map IdM groups to roles for auditors, admins, super-admins, etc.
    • Choose from the sidebar "Settings -> Configure".
    • Choose "Access control -> Groups".
    • Click the configuration drop down in the top left and choose "+ Add new group".
    • appliance-ui-img-2
    • On the groups screen, check the box that says "Look Up LDAP Groups".
    • Populate the "User to Look Up" box with a user who is in your super administrator group equivalent from IdM (probably "admins").
    • Click "Retrieve" beside the name.
    • A new drop down appears at the top of the screen titled "LDAP Groups for User". Click it and choose your administrator group.
    • Type a description of this group. "Admins" or something similar is fine.
    • Choose a role to map to this group, such as "EvmRole-super_administrator".
    • Choose the project/tenant this group and role will be associated with.
    • Click "Add" at the bottom of the screen to save the changes.
    • Repeat this for each IdM group you want to map to a CloudForms role.
    • appliance-ui-img-3
  4. Create and install your server certs and CAs (instructions assume you have a valid server cert and CA bundle file/cert).
    • In this example, idm-root-ca.cer will be the CA certificate from your IdM server.
    • Run the following commands to back up your self-signed certs and import your custom certs.
      • # mv /var/www/miq/vmdb/certs/server.cer /var/www/miq/vmdb/certs/server.cer.bk
      • # cp /root/cloudforms.crt /var/www/miq/vmdb/certs/server.cer
      • # cp /root/idm-root-ca.cer /var/www/miq/vmdb/certs/idm-root-ca.cer
      • # cp /root/cloudforms.key /var/www/miq/vmdb/certs/server.cer.key
    • Restart httpd.
      • # systemctl restart httpd
    • Ensure that your ssl configuration in the following step is updated to point to the newly copied /var/www/miq/vmdb/certs/idm-root-ca.cer as your SSLCACertificateFile. PKI will not work and client auth will fail until this step is performed.
  5. Update /etc/httpd/conf.d/manageiq-https-application.conf. 2
    • Update the virtual host with SSL options, SSLClientVerify require, certificate file locations, and OCSP options.
    • Add a permanent redirect from /dashboard/authenticate to https://<yourcloudformshostname>/dashboard/kerberos_authenticate.
    • The most important lines to add/note are the SSLOptions, SSLVerifyClient require, Require valid-user, SSLCertificateFile, SSLCACertificateFile, SSLOCSPEnable, SSLOCSPDefaultResponder, SSLOCSPResponderOverride, SSLUserName, and all of the RequestHeader directives.
    • Note: Please verify before using any SSLOCSP directives that your OCSP responder is available via an http connection (not HTTPS) as expected. If it isn't, please comment all "SSLOCSP" lines from your configuration.
    • Add %{SSL_CLIENT_S_DN_CN}x to the CustomLog line to add attribution by user for every SSL request.
    • Note: If you have a server certificate signed by a public CA, please add a line for the "SSLCertificateChainFile" and point it to the public CA's certificate bundle. The "SSLCACertificateFile" will still be populated with the IdM CA certificate since it will be validating the client certificates.
    • Note: If you copy and paste the /etc/httpd/conf.d/manageiq-https-application.conf file from the footnote of this post, or if you pull the one from the git repository, be sure to replace all references to "cloudforms.example.home" and "idm.example.home" with your CloudForms and IdM hostnames.
  6. Update /etc/httpd/conf.d/manageiq-external-auth.conf. 3
    • Update location and locationmatch settings for /dashboard/kerberos_authenticate with SSLVerifyClient require and Require valid-user.
    • Comment the entire location section for /dashboard/authenticate.
    • Update the api location block by commenting the line that says allow_admin and comment all basic PAM auth entries (to disallow sneaky text based logins and fully disable the default admin account).
    • Please note that this file is overwritten each time that external authentication is configured in the appliance_console.
    • After you're finished editing this file, please be sure to save a copy to /etc/httpd/conf.d/manageiq-external-auth.conf.bak or somewhere similar.
    • Note: After making changes to the httpd config files in steps 5 and 6, you must restart httpd for the changes to be recognized.
    • Note: Please test your PKI authentication at this point. If you are able to successfully log in using a client certificate, please skip Step 7 and move on to Step 8.
  7. Troubleshooting.
    • Ensure selinux rules are configured for LDAP queries and sssd-dbus (not necessary in most cases).
      • This is automatically configured by the appliance_console, but if it doesn't work properly you can fix this with the following commands: # setsebool -P allow_httpd_mod_auth_pam on and # setsebool -P httpd_dbus_sssd on.
    • If the groups being pulled via httpd from the your certificate information are not populating group names appropriately, go to "advanced configuration" in the UI (or edit settings.yml in the CLI) and ensure that your ldaphost entry is set correctly. You will see errors for this type of issue in /var/www/miq/vmdb/log/evm.log.
  8. Testing
    • Test your configuration by creating multiple users in IdM and attempting to authenticate with one of each of the following:
      • A user in the admins group (or the group you mapped to EvmRole-super_administrator)
      • A user in a group associated with a different EvmRole
      • A user in multiple groups associated with EvmRoles (in CloudForms clicking the top right drop down with the user's name should have an option to change groups for users in more than one group)
        • pki-multi-groups-img
      • A user with a revoked certificate (to test OCSP and the expected SSL handshake failure; this is a demonstration of an authentication failure)
        • pki-revoked-cert-img
      • A user NOT in a group mapped to an EvmRole (this is a demonstration of an authorization failure)
        • pki-user-not-in-group-img

Example configuration files are available in git at, as well as in the footnotes below.

  1. Example certificate fold command below:

  2. [root@idm ~]# echo "MIIEBjCCAu6gAwIB...M=" | fold -w 64 > john.crt
  3. Example /etc/httpd/conf.d/manageiq-https-application.conf file below:

  4. ## ManageIQ SSL Virtual Host Context  
     <VirtualHost *:443>  
      SSLOptions +StdEnvVars +ExportCertData  
     KeepAlive on  
     DocumentRoot /var/www/miq/vmdb/public  
     Redirect permanent /dashboard/authenticate https://cloudforms.example.home/dashboard/kerberos_authenticate  
     # The following redirects files must be included to  
     # handle most specific to least specific URLs.  
     Include conf.d/manageiq-redirects-ws  
     Include conf.d/manageiq-redirects-ui  
     Include conf.d/manageiq-redirects-websocket  
     Include conf.d/manageiq-external-auth.conf  
     ProxyPreserveHost on  
     RequestHeader set X_FORWARDED_PROTO 'https'  
     RequestHeader unset X_REMOTE_USER  
     RequestHeader set X_REMOTE_USER %{REMOTE_USER}e env=REMOTE_USER  
     RequestHeader set X-Remote-User %{SSL_CLIENT_S_DN_CN}s  
     ErrorLog /var/www/miq/vmdb/log/apache/ssl_error.log  
     TransferLog /var/www/miq/vmdb/log/apache/ssl_access.log  
     LogLevel warn  
     SSLEngine on  
     SSLProtocol all -SSLv2 -SSLv3  
     SSLCertificateFile /var/www/miq/vmdb/certs/server.cer  
     SSLCertificateKeyFile /var/www/miq/vmdb/certs/server.cer.key  
     SSLCACertificateFile /var/www/miq/vmdb/certs/idm-root-ca.cer  
     SSLVerifyClient require  
     SSLOCSPEnable on  
     SSLOCSPDefaultResponder http://idm.example.home/ca/ocsp  
     SSLOCSPOverrideResponder on  
     <Location /assets/>  
      Header unset ETag  
      FileETag None  
      ExpiresActive On  
      ExpiresDefault "access plus 1 year"  
     <Files ~ "\.(cgi|shtml|phtml|php3?)$">  
      SSLOptions +StdEnvVars +ExportCertData  
     <Directory "/var/www/cgi-bin">  
      SSLOptions +StdEnvVars +ExportCertData  
     <Location /proxy_pages/>  
      ErrorDocument 403 /error/noindex.html  
      ErrorDocument 404 /error/noindex.html  
     SetEnvIf User-Agent ".*MSIE.*" \  
      nokeepalive ssl-unclean-shutdown \  
      downgrade-1.0 force-response-1.0  
     CustomLog /var/www/miq/vmdb/log/apache/ssl_request.log \  
      "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x %{SSL_CLIENT_S_DN_CN}x \"%r\" %b"  
  5. Example /etc/httpd/conf.d/manageiq-external-auth.conf file below:

  6.  LoadModule authnz_pam_module modules/  
     LoadModule intercept_form_submit_module modules/  
     LoadModule lookup_identity_module modules/  
     LoadModule auth_kerb_module modules/  
     <Location /dashboard/kerberos_authenticate>  
     #  AuthType           Kerberos  
     #  AuthName           "Kerberos Login"  
     #  KrbMethodNegotiate On  
     #  KrbMethodK5Passwd  Off  
     #  KrbAuthRealms      EXAMPLE.HOME  
     #  Krb5KeyTab         /etc/http.keytab  
     #  Require            pam-account httpd-auth  
     # <Location /dashboard/kerberos_authenticate>  
      SSLVerifyClient require  
      require valid-user  
      ErrorDocument 401 /proxy_pages/invalid_sso_credentials.js   
     #<Location /dashboard/authenticate>  
     #  InterceptFormPAMService httpd-auth  
     #  InterceptFormLogin      user_name  
     #  InterceptFormPassword   user_password  
     #  InterceptFormLoginSkip  admin  
     #  InterceptFormClearRemoteUserForSkipped on  
     <LocationMatch ^/dashboard/authenticate$|^/dashboard/kerberos_authenticate$>  
       SSLOptions +StdEnvVars +ExportCertData -FakeBasicAuth  
       LookupUserAttr mail        REMOTE_USER_EMAIL  
       LookupUserAttr givenname   REMOTE_USER_FIRSTNAME  
       LookupUserAttr sn          REMOTE_USER_LASTNAME  
       LookupUserAttr displayname REMOTE_USER_FULLNAME  
       LookupUserGroups           REMOTE_USER_GROUPS ":"  
       LookupDbusTimeout          5000  
       SSLVerifyClient require  
       Require valid-user  
     <LocationMatch ^/api|^/vmdbws/wsdl|^/vmdbws/api>  
     #  SetEnvIf Authorization '^Basic +YWRtaW46' let_admin_in  
       SetEnvIf X-Auth-Token  '^.+$'             let_api_token_in    
     #  AuthType Basic  
     #  AuthName "External Authentication (httpd) for API"  
     #  AuthBasicProvider PAM   
     #  AuthPAMService httpd-auth  
       Require        valid-user  
       Order          Allow,Deny  
     #  Allow from env=let_admin_in  
       Allow from env=let_api_token_in  
       Satisfy Any   
       LookupUserAttr mail        REMOTE_USER_EMAIL  
       LookupUserAttr givenname   REMOTE_USER_FIRSTNAME  
       LookupUserAttr sn          REMOTE_USER_LASTNAME  
       LookupUserAttr displayname REMOTE_USER_FULLNAME   
       LookupUserGroups           REMOTE_USER_GROUPS ":"  
       LookupDbusTimeout          5000   
  7. Fake footnote to fix formatting.