Success Stories with Oracle Database Administration
Click HERE for Oracle APEX Administration Case Studies
Client: Leading independent, non profit research and development organization
The Oracle Database team at this organization is directly involved in NASA’s JUNO project, the exploration of Jupitor!
Project: Oracle Database Training for the JUNO Mission
SkillBuilders’ John Watson began teaching the Oracle DBA team at this Research Organization starting in 2014; the training and mentoring is ongoing and continues to this day. They’ve attended training in Oracle RAC, Security, Data Guard, Tuning, RMAN and more. (Additionally, developers in the organization have attended Java training.) More recently they’ve hired us in a consulting role to assist with, for example, tricky RMAN issues. Almost nine years of successful services delivered and counting!
Testimonial from Chris
“We have worked with John Watson during on-site engagements as well as numerous online classes. His depth of knowledge and his ability to teach complex topics is simply outstanding. Both my colleague and I are seasoned DBAs but we always gained a vast amount of new knowledge to be applied in our daily tasks and corporate operations.”
Client: Silicon Valley Power
Silicon Valley Power has provided the city of Santa Clara California with electricity for 125 years.
Project: Configure MAA on Usage/Billing Oracle Database
The primary project was configuring MAA (Maximum Availability Architecture): RAC-to-RAC Data Guard, between two ODAs (Oracle Data Appliances). Secondary projects included configuring Data Guard between two Windows systems, apply the ODA full stack patches, and setup an RMAN catalog database to manage backups for the whole environment. This was a lot of work for a short engagement, but we did manage to complete the project within budget. The result is a highly resilient environment for various databases, critical for the effective management of electricity usage and billing.
Client: Foremost Farms
Foremost Farms is a milk solids processing business owned by the dairy farm families who supply our milk. That’s what it means to be a cooperative – and it’s why our farmer-members have a vested interest in providing the highest-quality milk possible. They own Foremost Farms.
Project: Research running Oracle Data Appliance (ODA) in Virtualized Environments
This project was about investigating the ODA virtualized architecture and understanding how best to use it with single instance databases. Not because of RAC, because of licences.
You can run ODA as “bare metal” or “virtualized”. Typically, people who installed ODA more than, say, two years ago run bare metal: the two ODA nodes run normal Linux and the databases reside on ASM disc groups. This is all straightforward and should present no surprises to the DBA or the SA.
However, most people installing now follow Oracle’s recommendation and run it virtualized. The configuration can strike you as bizarre the first time you see it. The nodes run Xen Linux (Oracle’s version, of course) and everything happens in virtual machines. The databases reside on ACFS file systems, and are created from ACFS snapshots of preconfigured databases. The networking may take a while to get your head around.
In general, the virtualized platform is probably the way to go, but you do need to be a bit of a wizard with Grid Infrastructure and Xen to understand what is going on.
There are many GI facilities that can be used for fault tolerance and failover without RAC, which is not something that Is very well documented. Perhaps it should be.
Client: XIFIN
XIFIN is a health information technology company that leverages diagnostic information to improve the quality and economics of healthcare.
The XIFIN technology platform facilitates connectivity and workflow automation for accessing and sharing clinical and financial diagnostic data, linking healthcare stakeholders in the delivery and reimbursement of care.
Project: Configure New Oracle RAC Database Environment plus Migrations and Consolidations
SkillBuilders Oracle Certified Master DBA John Watson assisted a medical insurer to move several Oracle databases supporting OLTP, BI, and batch workloads from single instance 11.2.0.4 to a three node 12.1.0.2 RAC, consolidating the applications into just two databases as we did it. The destination platform was Sun servers with infiniband networking and EMC VMAX storage: this is a pretty big environment with demanding service level agreements. We trained the DBA and Systems Admin staff and provided further support during the implementation. Problems encountered, identified, and resolved included storage issues and performance problems, and (surprise!) a couple of Oracle bugs.
Client: Shoe Carnival
Shoe Carnival is one of the nation’s largest family footwear retailers with its own brand of style. Headquartered in Evansville, Indiana, Shoe Carnival stores are located in 35 states and Puerto Rico, as well as online at shoecarnival.com.
Project: Oracle Data Guard Implementation on RAC Exadata
SkillBuilders was initially asked to get the in-house DBA team up-to-speed on Oracle Data Guard. So, SkillBuilders’ Oracle Certified Master DBA John Watson taught our Data Guard Administration class over one week to the in-house DBA team. After the class wrapped up, Shoe Carnival IT management, aware of the time pressure and critical nature of the project, asked John to implement their first Data Guard configuration. Specifically, configuring one of their production databases for Data Guard and RAC (RAC -> RAC between two Exadata’s) and of course thorough testing including switch-over (failover) and switch-back.
Client: Kyocera (formerly SLD Laser)
SLD Laser – Brighter. Smaller. Safer. LaserLight delivered. The next evolution in lighting has arrived, shining brighter and farther than ever before. LaserLight surpasses the limits of LED to deliver safe, high luminance white laser light from an incredibly small light source. It’s enabling entirely new opportunities with SLD Laser’s GaN laser technology.
Project: Design and Implement Secure Channels for Hybrid Database Connections in AWS for 24×7 Multi-State Communications
Starting point: SkillBuilders had previously designed and implemented an Oracle APEX stack on Amazon AWS for the company’s internet-facing APEX applications. The company also has critical information in Microsoft SQLServer Databases.
Now we were tasked with constructing secure communication channels for Oracle and SQLServer connections between the company’s headquarters and three remote locations.
Our team configured a VPN Tunnel between three of the customer’s remote facilities and the AWS subnet where SkillBuilders had previously deployed their Web Tier and DB machines in a private Virtual Private Cloud (VPC). The environment consisted of a Customer Gateway which describes the onsite router (in this case Cisco), its external ip, software & version, and their LAN subnet; this is then connected to an AWS Virtual Private Gateway which describes the AWS LAN, which are connected with a “Site-to-Site VPC Connection”.
These VPN Tunnels provide fully encrypted, 24×7, two-way traffic, including sqlnet & odbc connections between Oracle and Microsoft SQLServer Databases.
Large Advertising & Marketing Services Company
The project: build a secure Oracle APEX environment on Oracle Cloud for the company’s internet-facing applications.
Starting point: one virtual machine running a pre-created database with APEX installed
End point: the full Apache/Tomcat/ORDS/APEX/database stack, SSL enabled
Budget: 20 hours
As with any cloud provider, it takes a while to understand how to use the cloud infrastructure, particularly the networking and security. Once you have got past that, you can start work.
First, the Oracle Enterprise Linux default configuration isn’t good enough. You need to install a graphical desktop, and also various RPMs and update OpenSSL. The firewall needs some configuration too.
Then the usual: download and install Tomcat; download ORDS, configure and deploy to Tomcat; download the Apache http server plus the APR libraries and compile. We added an OpenVPN server for SQL*Net and ssh access, so that only the Apache listening ports would be exposed to the internet. Everything configured for automatic start on boot up.
All done within budget.
Mid-Size American City
Designed and implemented Oracle Database Standard Edition Failover via intelligent use of scripting and various OVM and networking facilities.
Implemented OVM (Oracle Virtual Machine, Oracle’s take on Xen) at two sites, including a DR solution such that database VMs at either site can failover to the other in the event of problems with minimal data loss. Zero data loss and full automation were not possible as only Standard Edition licences were available, but intelligent use of scripting and various OVM and networking facilities give the client something very close to that. Each site had two physical machines: one (with magnetic discs) to run the OVM Manager and be an NFS server, the other (with mixed-use and write-intensive SSDs) to run the VMs to be used as database servers. Using NFS repositories, we directed backups and archives from each site to the other site. A minor complication was that for licence compliance we had to use hard partitioning to restrict the CPUs available to the database servers. In the event of a failure at one site, new VMs can be launched from templates at the surviving site and the databases restored onto them. Not Data Guard level resilience, but pretty darn good.
Brown University Administration
Since 2008, SkillBuilders has trained Brown University developers and Oracle Administrators in new skills vital to the Universities success. This includes Oracle Database performance tuning, new features, RMAN, Enterprise Manager / Grid Control and Grid Infrastructure. For developers SkillBuilders has provided education in Java Programming, Servlets, JSP, XML, Ruby on Rails and more.
Soraa
SkillBuilders designed and installed active standby using Oracle Database Standard Edition, which supports Soraa production line. SkillBuilders also provides Oracle licenses, in this case saving Soraa tens of thousands (perhaps hundreds) by advising customer Standard Edition would meet customer requirements.
We also implemented RMAN backups, Alert/Incident monitoring and general database tuning. We remain on-call for support issues.
North Carolina Department of Labor
In 2012, SkillBuilders provided the Department of Labor in North Carolina state government with a turnkey APEX configuration, i.e. everything they needed to start deploying internal and internet-facing APEX applications. This included database installation and patching, APEX and APEX listener configuration, Java container (Glassfish) installation and configuration and assistance with firewall updates.
SkillBuilders continues to provide Oracle Database, APEX and infrastructure support; patches, upgrades and troubleshooting.
Acxiom
SkillBuilders provides ongoing Oracle Database advanced administration training. This includes Oracle Data Guard, Administration and Performance Tuning training. One student, who provided a perfect 5.0 score in his post-course evaluation, exclaimed, “I really enjoyed John as my instructor. He was extremely professional, personal, and deeply concerned about us understanding the concepts of in this course.”
Oracle Corporation
SkillBuilders teaches 12c Database new features to Oracle Corporation employees. We developed a custom course and have delivered it 12 times and counting! We also teach other subjects such as Oracle Database Performance Tuning and Data Guard Administration.
Waters
When Waters Corp software customers needed a tool to manage ASM disks, non-ASM disk, RMAN backups, imports/exports, monitor and send alerts, manage database users (e.g. password management), SkillBuilders developed an integrated Oracle APEX solution. The “Waters Database Manager”, or WDM, also provides other general management functions such as job scheduling and reporting.
Stretched (Extended Distance) Clusters / ASM
In 2010, Oracle Master John Watson was the development DBA (designer / implementer) behind the successful implementation of a stretched cluster (aka extended distance / geo-cluster). Here’s the story….
We all know what happens if the database that tracks passengers through security fails: the queues grow longer, and within minutes they stretch half way round the terminal. The database that tracks the baggage is as critical: thousands of bags stack up when they should be on the conveyor belts.
At this client (a large airport in Africa) it appeared that even Data Guard would not be adequate to keep things moving. Fast Start Failover with the Data Guard Broker is fast: it can initiate within seconds. But most DBAs will want to build in a delay of a few minutes. Then it takes time actually to switch over to the standby and reconnect all the sessions. That would have been too slow, given the speed with which chaos would escalate in that environment. And it would need Enterprise Edition licences.
RAC looks like the answer: near instantaneous failover of services and sessions from one instance to another if you lose a node. But it doesn’t protect you against losing the site. Or does it? Yes, if you set up a stretched cluster. At each of two airport terminals, we had a database server and a storage array, connected through a fiber switch. A separate ethernet gave the terminals on the security desks and the baggage scanners access, load balanced across both server nodes. ASM handled the mirroring. Losing a server node (not uncommon given erratic power suppliers and unreliable networks) caused all the broken sessions to reconnect (yes, we automated that) to the surviving node with a break in service time of only seconds. When the node came back online, ASM would re-synchronize the database copies. It really worked.
And, perhaps best of all, it worked with Standard Edition licences.
Manual Failover with Standard Edition
In 2013 we configured a manual failover on Standard Edition for an LED light manufacturer. This new company uses their Oracle database to keep their production lines moving – but the cost of Enterprise Edition and Advanced Data Guard is prohibitive and unnecessary, since our manual failover implementation will keep them up and running with 15 minutes of a failure on the primary server.
Advanced Queueing and Resource Manager
In 2008, Oracle Certified Master Administrator John Watson used his knowledge of Oracle Database features to create a clever solution to a performance problem. The client was maxed at 48,000 transactions per hour; John used Advanced Queuing and Resource manager to quadruple that. Read on…
SMSs (i.e. texts) make money – if you can process them fast enough!
The volume of SMS traffic is unbelievable. This isn’t just people voting for a Simon Cowell show: it is requests for ringtones, wallpapers, jokes, games, who knows what else. Millions of SMSs, and they all generate income. Or they should.
At this client (a cellphone network operator in Africa) they were processing maximum 48000 requests per hour. More than that could not go through the database: the sessions just hanged. When the SMS aggregators were throwing messages at peak rate to the application servers, the queues would shoot out, and eventually the application servers would start dropping messages. Bad, very bad: that meant not just losing a sale, but probably losing a customer.
What is the answer to a scalability problem? RAC, of course. Well, not always. We investigated the transactions, and the problem was enqueue contention: the business model relied on a pyramid hierarchical sales structure, so concurrent transactions at a low level would block each other at a higher level. We had to explain: RAC can’t help with TX enqueues, and might make things worse. The answer: re-design the transactions to use Advanced Queueing. The low level updates put a message on a queue, and commit. Fast. Then de-queueing every few seconds would process a group of a several hunded updates as one logical transaction. Problem solved: now processing up to 160000 requests per hour.
But they wanted more. Rolling out a banking application for transferring funds from one subscriber account to another, all with SMSs, looked like doubling the request rate. And the box was already running at around 90% utilization. It was trime to look at processing priorities, and what else was running on that box. The “what else” was reports. Such as analyzing the effect on sales of a radio advert – in real time. This is important and the information needs to be timely, but does it really matter if the analysis take three minutes instead of thirty seconds? This is where the Resource Manager comes into its own. Not widely used and understood (a lazy DBA can just throw more hardware and Oracle licences at the problem) but in this case perfect. By setting up appropriate priorities, every sales request would respond fast no matter what the workload, and at times the reports would degrade.
Just another example of how enabling a couple of advanced features can save the need for more hardware and more licences.